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EXECUTIVE  SUMMARY 


Test  schedule  development  is  a  specialized  process  that  is  complex,  time- 
consuming,  and  iterative.  For  Department  of  Defense  program  management 
offices,  test  schedules  play  a  critical  role  in  program  schedule  development  and 
decision  making.  This  research  captures  a  Department  of  Defense  program 
management  office’s  existing  test  scheduling  process  that  is  developed  based  on 
heuristics.  This  research  establishes  requirements  for  conversion  of  the  test 
scheduling  process  into  a  test  scheduling  optimization  model  that  is  constraint 
and  rule-based. 

The  initial  problem  formulation  is  generated  in  conjunction  with  Dr.  Gerald 
Brown,  NPS  Distinguished  Professor  of  Operations  Research.  Operational  needs 
are  developed  based  on  the  test  scheduling  process  and  based  on  the  problem 
formulation.  An  analysis  of  alternatives  is  performed  against  existing  optimization 
models  using  the  operational  needs.  Detailed  requirements  are  generated  based 
on  the  critical  operational  issues,  based  on  the  operational  needs,  and  based  on 
the  problem  formulation. 

Another  thesis  student,  Shane  A  Edwards,  (Edwards  2015)  finalizes  the 
problem  formulation  in  conjunction  with  Dr.  Brown,  develops  the  optimization 
model,  and  performs  his  own  developmental  optimization  model  testing. 

The  final  model  run  provided  by  Shane  Edwards  is  reformatted  into  3- 
dimensional  daily  test  schedules  for  the  five  different  test  schedules  generated  by 
the  model.  The  model  input  files,  output  files,  and  reformatted  3-dimensional 
daily  schedules  are  used  to  verify  conformance  to  the  detailed  requirements. 

The  3-dimensional  daily  test  schedules  are  aggregated  into  3-dimensional 
monthly  test  schedules.  The  reformatted  optimization  model-provided  monthly  3- 
dimensional  schedules  are  assessed  against  the  test  and  evaluation  heuristic  3- 
dimensional  schedules.  Assessment  results  are  validated  against  the  operational 


XXI 


needs,  the  critical  operational  issues,  and  for  operational  effectiveness  and 
suitability. 

Results  show  that  the  optimization  model  developed  by  Shane  Edwards 
(Edwards  2015)  meets  the  majority  of  the  requirements  and  provides  schedules 
that  are  reasonably  close  to  what  the  test  and  evaluation  planners  would  use. 
The  results  of  this  research  indicate  that,  with  additional  work  to  make  the  input 
more  user-friendly  and  to  display  the  output  differently,  the  test  and  evaluation 
test  schedule  optimization  model  would  be  a  good  tool  for  the  test  and  evaluation 
schedulers. 

Reference 

Edwards,  Shane  A.  2015.  “Optimizing  Department  of  Defense  Acquisition 
Development  Test  and  Evaluation  Scheduling.”  Masters  thesis,  Naval 
Postgraduate  School. 
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I.  INTRODUCTION 


Test  schedule  development  is  a  specialized  process  that  is  complex,  time- 
consuming,  and  iterative.  For  Department  of  Defense  (DOD)  program 
management  offices  (PMOs),  test  schedules  play  a  critical  role  in  program 
schedule  development  and  decision  making.  This  research  captures  a  DOD 
PMO’s  existing  test  scheduling  process  that  is  developed  based  on  heuristics. 
This  research  establishes  requirements  for  conversion  of  the  test  scheduling 
process  into  a  test  scheduling  optimization  model  that  is  constraint  and  rule- 
based.  The  developed  model  is  verified  and  validated  to  assess  whether  it  is 
functioning  as  intended  against  the  requirements  and  to  assess  the  model 
against  heuristic  schedules.  The  purpose  of  this  research  is  to  determine  if  the 
test  scheduling  optimization  model  can  be  used  to  aid  test  planners  in  their  test 
planning  schedule  development  efforts. 

Stakeholders  in  this  research  are  DOD  PMOs  and,  specifically,  test 
personnel  supporting  DOD  PMOs.  Potential  interest  could  include  developmental 
and  operational  test  agencies,  and  contractors  who  need  to  perform  test  planning 
activities. 

To  understand  what  influences  defense  system  acquisition  test 
scheduling,  a  basic  appreciation  is  needed  of  DOD  organizational  relationships 
and  of  DOD  processes.  Department  of  Defense  processes  of  significance  include 
budgeting,  system  acquisition,  and  requirements  development  and  verification 
processes.  Providing  information  in  these  areas  provides  context  for  PMO  test 
scheduling  activities  and  promotes  understanding  of  the  many 
interdependencies. 

A.  ORGANIZATIONS  AS  SYSTEMS 

Department  of  Defense  system  acquisition  is  performed  by  systems 
command  organizations  within  each  of  the  services.  These  systems  command 
organizations  exist  to  acquire  needed  warfighter  capabilities  in  the  form  of 
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systems.  The  ultimate  goal  of  the  systems  acquisition  community  is  to  acquire 
systems  that  will  ensure  warfighters  achieve  their  mission  objectives.  Program 
management  offices  within  systems  commands  are  the  organizations  that 
perform  the  system  acquisition  work. 

As  Anderson  and  Johnson  discuss,  organizations  are  themselves  systems 
(1997,  2).  This  organizational  system  thinking  is  synopsized  in  Figure  1  and  is 
applicable  to  systems  commands. 


Direction 


Inputs 

t  I 


Throughput 

f 


■> 


Results 


Figure  1.  Organizational  Systems  Framework 


Figure  1  and  the  associated  discussion  is  adapted  from  class  notes  and 

lectures  and  is  based  on  data  provided  by  the  NPS  instructor  Cary  Simon  (Nancy 

C.  Roberts,  class  notes  provided  by  Cary  Simon  based  on  slide  generated  by 

Roberts  in  2000,  class  notes  and  lectures  provided  to  author,  September  12, 

2012).  Figure  1  depicts  organizations  as  systems  with  internal  and  external 

dependencies.  The  organization  itself  is  impacted  by  external  environment,  by  its 

context  within  that  environment,  by  key  success  factors,  and  by  organizational 
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direction,  which  are  all  considered  input.  External  environments  to  an 
organization  are  political,  economic,  social,  and  technological.  Key  success 
factors  indicate  what  it  will  take  for  the  organization  to  be  successful.  System 
direction  is  influenced  by  mission,  values,  mandates,  strategic  issues,  vision, 
goals,  and  strategies.  The  organization  itself  has  internal  components  that  impact 
the  performance  of  the  organization,  shown  as  “throughput.”  These  internal 
components  are  people,  tasks/jobs,  structure,  processes,  technology,  and 
culture.  Results  of  the  organization  consist  of  outputs  and  outcomes.  Outputs  are 
what  the  organization  delivers  such  as  goods  and  services.  Outputs  can  be 
measured  in  terms  of  performance  relative  to  the  key  success  factors.  Outcomes 
can  be  intended  or  unintended,  and  are  the  consequences  of  the  outputs  viewed 
in  context  with  the  external  environment. 

Program  management  offices  can  be  considered  organizational  systems 
and  can  be  looked  at  from  this  organizational  systems  framework  viewpoint,  with 
inputs,  outputs,  and  outcomes.  Understanding  a  PMO  as  an  organization  with 
external  and  internal  dependencies  furthers  understanding  of  the  impacts  of 
these  dependencies  on  test  schedules. 

B.  DOD  DECISION  SUPPORT  SYSTEMS  TRIAD 

As  shown  in  Figure  1 ,  to  perform  system  acquisitions,  the  people  in  a 
PMO  organization  must  perform  tasks  that  will  result  in  the  intended  outcome.  The 
tasks  performed  are  bound  by  the  external  environment  of  an  imposed  set  of  DOD 
acquisition  processes.  As  described  by  Defense  Acquisition  University  (DAU) 
(2017b),  for  a  DOD  PMO,  these  external  decision  support  processes,  also  known 
as  the  Big  “A,”  are  the  DOD  decision  support  systems  triad  of:  the  planning, 
programming,  budgeting  and  execution  (PPBE)  process,  the  joint  capabilities  and 
development  system  (JCIDS)  process,  and  the  defense  acquisition  system 
process.  As  described  on  DAU’s  defense  acquisition  portal,  the  “[Department  of 
Defense]  has  three  principal  decision-making  support  systems.  Together,  the 
systems  provide  an  integrated  approach  to  strategic  planning,  capabilities  needs 
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assessment,  systems  acquisition,  and  program  and  budget  development”  (DAU 
2017b).  These  three  processes  are  symbiotic,  as  shown  in  Figure  2. 


CJCSI  5123.01 

CJCS  3170.01 

VCJCS/JROC 

Oversight 


Milestone  Decision  Oversight 

Authority  (MDA) 

Oversight 


Figure  2.  DOD  Decision  Support  Systems  Triad.  Source:  DAU 

(2017a). 


Using  the  Anderson  and  Johnson  method  of  relationship  depiction  (1997,  2), 
the  DOD  decision  support  systems  triad  and  the  triad  stakeholders  are  imposed 
upon  the  PMO,  as  shown  in  Figure  3. 
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Figure  3.  Inter-relationships  and  Intra-relationships  between 

Stakeholders,  the  DOD  Decision  Support  Systems  Triad, 

and  the  PMO 


Department  of  Defense  components  are  defined  by  the  Undersecretary  of 
Defense  for  Acquisition  Technology  and  Logistics  (USD  (AT&L))  as  “[Office  of  the 
Secretary  of  Defense],  the  Military  Departments,  the  Office  of  the  Chairman  of 
the  Joint  Chiefs  of  Staff  and  the  Joint  Staff,  the  Combatant  Commands,  the 
Office  of  the  Inspector  General  of  the  Department  of  Defense,  the  Defense 
Agencies,  the  DOD  Field  Activities,  and  all  other  organizational  entities  within  the 
DOD”  (USD  (AT&L)  2017,  1). 

Taking  the  concept  provided  in  Figure  1,  we  expand  upon  this  in  Figure  3, 

adding  in  information  from  Figure  2,  showing  the  specifics  of  the  PMO  within  the 

DOD  triad.  This  relationship  diagram  shows  that,  in  addition  to  the  many  internal 

organizational  inter-relationships,  there  are  many  external  stakeholders  who  will 

influence  the  resulting  system  acquisition  strategy,  including  the  three  symbiotic 
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and  intertwined  process  triad.  The  external  success  factors  of  the  systems 
acquisition  strategy  include  political,  economic,  social,  technological,  mission, 
values,  mandates,  strategic  issues,  visions,  goals,  and  strategies. 

1.  Planning,  Programming,  Budgeting  and  Execution  Process 

The  PPBE  process  is  a  fiscal  year  calendar-based  process  that  is  used  to 
plan,  budget,  and  execute  a  DOD  acquisition  program  (DAU  2017c).  Details  of 
the  PPBE  process  are  contained  in  Department  of  Defense  directive  (DODD) 
7045.14.  As  explained  in  this  Under  Secretary  of  Defense  (Comptroller)  (USD  (C) 
2013)  directive,  the  PPBE  process  is  used  to  identify  resource  requirements, 
allocate  resources,  produce  a  five-year  programming  plan,  and  budget  and 
execute  a  yearly  budget.  PPBE  results  must  align  with  the  needs  identified  in  the 
national  security  strategy,  while  being  constrained  by  resources  allocated  by 
Congress  (USD  (C)  2013,  2). 

2.  Joint  Capabilities  and  Development  System  Process 

The  JCIDS  process  is  a  strategic,  calendar-based,  and  event-based 
process  that  assesses  strategic  needs  in  conjunction  with  critical  needs  identified 
through  the  PPBE  process  due  to  real  world  events.  Details  of  the  JCIDS 
process  are  contained  in  Chairman  of  the  Joint  Chiefs  of  Staff  instruction  (CJCSI) 
3170.01  (CJCS  2015b)  and  in  the  supporting  JCIDS  manual  (CJCS  2015c). 
What  follows  in  this  paragraph  is  a  high  level  explanation  of  JCIDS  activities 
discussed  in  these  CJCS  documents.  The  JCIDS  process  identifies  DOD  gaps 
that  need  to  be  filled  (CJCS  2015a,  A-1).  Before  a  new  or  modified  system  is 
proposed,  the  JCIDS  process  explores  changing  Doctrine,  Organization, 
Training,  Materiel,  Leadership  and  education,  Personnel,  and  Facilities 
(DOTMLPF)  (CJCS,  2015b,  2).  If  a  need  is  deemed  to  be  a  priority,  an  analysis 
of  alternatives  (AoA)  is  conducted  to  determine  whether  an  existing  system  can 
be  modified  to  meet  the  need  or  a  new  system  is  needed  (USD  (AT&L)  2017, 
18).  The  output  of  the  JCIDS  process  is  a  capability  document  that  provides 
warfighter  requirements  for  system  modification  or  for  a  new  systems  acquisition 
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(CJCS  2015b,  2).  The  capability  document  is  provided  to  the  systems  command 
PMO  for  systems  acquisition  execution. 

3.  Defense  Acquisition  Process 

The  defense  acquisition  process  is  an  event-based  process  that  guides 
program  managers  and  DOD  components  through  milestones,  decision  points, 
and  phases  for  the  lifecycle  of  a  system’s  acquisition,  sustainment,  and  disposal. 
DODD  5000.01  (USD  (AT&L))  2003)  and  Department  of  Defense  instruction 
(DODI)  5000.02  (USD  (AT&L))  2017)  provide  details  of  defense  acquisition 
policy,  process,  and  procedures.  Systems  are  given  acquisition  category  (ACAT) 
levels  that  indicate  required  documentation  and  activities  (USD  (AT&L))  2017,  3). 

Upon  determination  that  there  will  be  a  system  modification  or  new  system 
acquisition  based  on  the  JCIDS  process,  the  program  manager  and  the 
supporting  PMO  personnel  then  start  the  arduous  task  of  working  through  the 
defense  acquisition  process.  The  PMO  formulates  a  program  and  develops  the 
required  systems  acquisition  documentation  in  support  of  the  acquisition 
approach  (USD  (AT&L)  2017,  7). 

C.  SYSTEMS  ACQUISITION  PROGRAM  STRUCTURE 

As  shown  by  the  information  provided,  the  DOD  decision  support  systems 
triad  is  event  driven  and  calendar  driven,  which  further  complicates  PMO 
activities  since  the  PMO  must  balance  and  support  all  three  processes.  The 
management  framework,  within  which  a  systems  acquisition  program  exists,  is 
called  the  systems  acquisition  program  structure  (DODI  2000.02  2017,  6). 

The  systems  acquisition  program  structure  guides  the  PMO  by  providing  a 
general  approach  to  systems  acquisition  in  the  form  of  phases  and  events  as 
shown  in  Figure  4  (DODI  2000.02  2017,  9). 
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Figure  4.  System  Acquisition  Program  Structure  (Hardware 
Dominant).  Source:  DODI  5000.02  (2017). 


Figure  4  shows  a  systems  acquisition  program  structure  for  a  hardware 
dominant  system  (DODI  5000.02  2017,  9).  What  follows  in  this  section  is  an 
explanation  of  Figure  4  based  on  the  DOD  instruction.  The  systems  acquisition 
program  structure  identifies  five  phases  that  a  system  may  go  through  (material 
solution  analysis,  technology  maturation  and  risk  reduction,  engineering  and 
manufacturing  development,  production  and  deployment,  and  operations  and 
support).  A  system  can  enter  this  program  structure  at  any  of  the  three 
milestones  (A,  B,  or  C),  depending  on  the  maturity  of  the  technology  used  in  the 
system  and  depending  on  the  maturity  of  the  system  itself  (USD  (AT&L)  2017,  6). 
This  means  that  a  highly  mature  system  may  enter  the  defense  acquisition 
program  structure  at  milestone  B  and  only  go  through  three  phases:  engineering 
and  manufacturing  development,  production  and  deployment,  and  operations 
and  support.  A  system  goes  through  many  decision  points  to  assess  status  and 
to  make  needed  decisions,  as  indicated  by  a  yellow  diamonds  (USD  (AT&L) 
2017,  6).  The  milestone  triangles  at  the  top  of  Figure  4  are  also  decision  points. 


8 


DODI  5000.02  (2015)  provides  variations  of  this  program  structure  based 
on  aspects  of  particular  programs  that  may  require  different  focus,  such  as 
acquisition  of  software  intensive  systems,  accelerated  acquisition  systems,  and 
hardware  dominant  systems.  The  Figure  4  diagram  is  for  a  hardware  dominant 
system  (DODI  5000.02  2015,  9).  The  intent  of  providing  many  options  in  this 
instruction  is  to  move  away  from  a  one-size-fits-all  checklist  mentality  in  the 
acquisition  community  and  to  give  program  managers  the  flexibility  to  provide  a 
cost  and  schedule  optimized  solution  as  described  in  better  buying  power  (BBP) 
3.0  (USD  (AT&L)  2015).  The  seven  BBP  3.0  (USD  (AT&L)  2015)  principles  are: 
achieve  affordable  programs,  control  costs  throughout  the  product  life  cycle, 
incentivize  productivity  and  innovation  in  industry  and  government,  eliminate 
unproductive  processes  and  bureaucracy,  promote  effective  competition, 
improve  tradecraft  in  acquisition  of  services,  and  improve  the  professionalism  of 
the  total  acquisition  workforce. 

Flexibility  in  approach  to  the  systems  acquisition  program  structure  is 
highlighted  by  the  dotted  lines  between  the  acquisition  phases  and  the  color 
variations  in  Figure  4,  as  this  is  a  significant  departure  from  prior  versions  of  this 
program  structure  that  contained  solid  lines  and  colors.  As  stated  in  the 
instruction,  “[Milestone  decision  authorities]  should  tailor  regulatory  procedures  in 
the  document  consistent  with  sound  business  practice  and  the  risks  associated 
with  the  product  being  acquired”  (DODI  5000.02  2015,  2). 

To  support  the  DOD  decision  support  systems  triad,  the  PMO  must 
perform  many  scheduling  activities.  Although  PMO  schedules  have  similarities, 
all  PMO  schedules  are  unique.  As  the  PMO  performs  planning  and  decision 
making  activities,  they  assess  different  courses  of  action  (COAs).  The  PMO 
varies  program  entry  points,  milestones,  numbers  of  systems  to  be  delivered  in 
each  phase,  delivery  schedules,  test  events,  and  test  schedules.  The  PMO  puts 
together  an  achievable,  cost-constrained,  best-value  plan  for  acquisition  of  the 
system.  The  chosen  schedule  is  then  incorporated  into  the  acquisition 
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documentation,  including  the  test  and  evaluation  master  plan  (TEMP)  (USD 
(AT&L)  2017,  106). 

As  the  PMO  progresses  through  execution  of  their  plan,  and  as  internal 
and  external  events  change,  the  PMO  constantly  reassesses  the  schedule  and 
performs  course  corrections.  This  means  that  the  schedule  is  in  constant 
scrutiny,  and  that  schedule  and  cost  COAs  are  in  constant  development 
throughout  the  systems  acquisition  and  sustainment  life  cycle. 

D.  SYSTEMS  ENGINEERING  PROCESS 

As  explained  by  DAU  (2014),  the  systems  engineering  process  is  an 
internal  DOD  organizational  process  that  provides  the  technical  framework  within 
the  DOD  systems  acquisition  program  structure  to  guide  the  PMO  during 
systems  acquisition.  What  follows  in  this  section  is  an  explanation  of  the  DOD  DAU 
systems  engineering  process.  Within  the  PMO,  the  systems  engineer  is 
responsible  for  the  definition  and  for  the  execution  of  the  specific  systems 
engineering  aspects  of  the  PMO  acquisition  plan  and  strategy.  While  each 
systems  acquisition  is  different,  the  systems  engineering  process  remains  the 
same,  although  complexity,  technical  reviews,  timeframes,  and  responsible 
parties  may  vary.  The  DAU  defines  the  systems  engineering  process  in  the  form 
of  a  “V,”  as  is  shown  in  Figure  5  (DAU  2014).  The  Figure  5  systems  engineering 
“V”  shows  the  path  from  operational  need  identification  to  fielding  a  capability  to 
meet  that  need. 


10 


Systems  Engineering 


Operational 
Need 


c 


Delivered 


Capability  /lOC/FOC 


o 

o 


\ 


Requirements 


<=> 


Validated 

Solution 


<>  OT&E 


Technical  Processes 


Stakeholder 
Requirements 
Definition 
Requirements 
Analysis 
Architecture 
Xj-  Design 


Decision  Analysis 
Technical  Planning 
■  Technical  Assessment 


% 


Design  Product/ 


© 


/ 

DT&E 


1$ 


Technical  Processes 


Transition 
Validation 
Verification 
Integration 
Implementation 


Technical  Management  Processes 


■  Requirements  Management 

■  Risk  Management 
■Configuration  Management 


•Technical  Data  Management 
■  Interface  Management 


Enables  a  balanced  approach  for  delivering  capability  to  the  warfighter 


Figure  5.  System  Engineering  Process.  Source:  DAU  (2014). 


If  there  is  deemed  to  be  a  need  for  a  system  capability,  and  the  need  is 
considered  to  be  a  priority  within  the  budget,  a  program  is  begun  within  the  DOD 
with  the  development  of  the  operational  need  via  a  capability  document  (CJCS 
2015b,  A-1 0). 

The  system  engineer,  in  concert  with  PMO  personnel,  performs  market 
surveys  and  requests  for  information  from  industry  in  order  to  hone  in  on  what  is 
currently  available  in  the  marketplace  that  might  meet  the  need  (USD  (AT&L) 
2017,  88).  The  systems  engineer  considers  whether  an  existing  system(s)  could 
be  modified  to  meet  the  need.  Simultaneous  with  these  activities,  a  formal  AoA  is 
performed  to  see  if  there  are  existing  systems  within  the  DOD  that  might  meet 
the  need  and  to  determine  the  correct  type  of  system  (USD  (AT&L)  2017,  19). 
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The  system  engineer,  in  conjunction  with  PMO  personnel,  will  work  through 
potential  alternatives  to  meet  the  need. 

What  follows  on  the  next  several  pages  is  a  detailed  explanation  of  the 
flow  through  the  systems  engineering  “V”  diagram  provided  in  Figure  5  (DAU 
2014).  The  systems  engineering  “V”  starts  with  user  capabilities  identified  by  the 
end  user  shown  as  “operational  needs.”  The  JCIDS  process  results  in  the 
capabilities  document  that  starts  the  systems  engineering  process.  The  JCIDS 
process  involves  the  PMO  in  order  to  assess  whether  the  capabilities  defined  are 
achievable  and  to  determine  the  priority  of  the  stakeholder  needs  (USD  (AT&L) 
2017,  21).  The  systems  engineer  works  with  the  user  representative  to  elicit 
stakeholder  prioritized  requirements,  taking  into  account  affordability,  product 
and  technology  availability,  and  technology  readiness  levels  (USD  (AT&L)  2017, 
20).  The  answers  to  these  questions  translate  into  capability  and  program  risk 
(USD  (AT&L)  2017,  19).  This  technology  risk  level  will  guide  specifics  of  the 
program  acquisition  plan,  contract  type,  and  entry  point  into  the  acquisition 
program  structure. 

The  user  representative  identifies  threshold  and  objective  capabilities  in 
the  form  of  a  capability  document,  which  defines  operational  needs  (CJCS 
2015c,  D-A-1).  The  capability  document  is  augmented  by  a  mission  profile 
document,  which  explains  how  the  system  operates  in  the  mission  environment 
in  peacetime  and  in  wartime  (CJCS  2015c,  B-24).  The  capability  document  is 
further  augmented  by  a  DOD  architecture  framework  (DODAF)  set  of  views  that 
explain  how  the  system  operates  within  the  communications  architecture  (in 
support  of  joint  interoperability)  and  that  provide  specific  details  underpinning  the 
net  ready  key  performance  parameter  (NR  KPP)  (CJCS  2015c,  C-B-1).  A  KPP 
within  a  capability  document  is  deemed  to  be  a  high  priority  capability.  The  NR 
KPP  is  a  priority  capability  that  all  DOD  systems  must  meet  in  order  to  be  fielded 
(CJCS  2015c,  D-61).  The  capability  document  is  updated  at  program  milestones 
as  the  systems  acquisition  proceeds  through  the  systems  acquisition  program 
structure. 
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The  systems  engineering  “V”  shown  in  Figure  5  continues  with 
decomposition  of  the  user  capabilities  document,  mission  profile  document,  and 
DODAF  views  into  system  requirements.  Requirements  are  also  incorporated 
from  DOD  instructions,  directives  and  regulations.  The  resulting  system 
performance  specification  is  the  document  that  is  provided  to  the  organization 
that  is  responsible  for  the  system  design  (USD  (AT&L)  2017,  23).  In  most  cases, 
the  organization  that  designs  the  system  consists  of  an  outside  contractor  or 
contractors. 

The  PMO  develops  a  request  for  proposal  (RFP)  to  elicit  potential  system 
product  developer  solutions  to  the  operational  needs  specified  in  more  detail  in 
the  system  performance  specification.  Upon  completion  of  all  required  systems 
acquisition  documents,  upon  congressional  approval  of  the  program  budget,  and 
upon  approval  by  the  Secretary  of  Defense,  the  RFP  is  released  to  potential 
contractors  (USD  (AT&L)  2017,  7).  RFP  responses  are  evaluated  using  the 
source  selection  process  based  on  source  selection  criteria.  Once  the  system 
product  developer(s)  are  selected  by  the  source  selection  authority,  a  contract,  or 
contracts,  is  awarded  (USD  (AT&L)  2017,  25). 

The  systems  engineering  “V”  shown  in  Figure  5  continues  with  product 
development.  The  system  product  developer(s)  proceed  through  product  design 
and  development,  and  through  the  systems  engineering  technical  review  gates  of 
the  systems  acquisition  process  (USD  (AT&L)  2017,  26).  The  system  product 
developer  performs  system  requirements  analysis,  architecture  design,  design 
implementation,  system  integration,  and  system  test,  and  delivers  the  developed 
system  to  the  PMO  (USD  (AT&L)  2017,  26). 

The  TEMP  provides  a  high  level  coordinated  test  plan  within  the  approved 

acquisition  plan,  acquisition  strategy,  and  overarching  program  schedule,  and  is 

included  as  one  of  the  required  acquisition  documents  (USD  (AT&L)  2017,  4). 

Developmental  testing  (DT)  and  operational  testing  (OT)  test  plans  are 

articulated  at  a  high  level  in  the  TEMP.  The  capability  document,  the  mission 

profile,  and  the  DODAF  are  used  in  development  of  the  DT  and  OT  test  plans  by 
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the  DT  and  OT  community.  The  TEMP  is  updated  at  program  milestones  within 
the  specific  system  acquisition  program  structure  and  as  supported  in  the 
acquisition  plan  and  the  acquisition  strategy. 

The  systems  engineering  “V”  shown  in  Figure  5  continues  with  solution 
verification  and  validation.  Upon  receipt  of  the  system,  the  PMO  performs 
developmental  verification  through  inspection,  demonstration,  certification, 
analysis,  and  testing.  The  system  performance  specification  is  used  to  develop 
the  DT  program  that  is  used  by  the  PMO  to  verify  that  the  system  meets  the 
system  performance  specification  requirements.  The  PMO  assesses  the 
probability  of  success  in  OT  and  in  meeting  system  capabilities. 

Upon  completion  of  PMO  DT  verification  activities,  Figure  5’s  systems 
engineering  “V”  continues  with  delivery  to  the  operational  test  community  who 
provide  validation  against  the  capability  document  (USD  (AT&L)  2017,  25).  The 
operational  test  agency  (OTA)  will  assess  the  system  for  operational  effectiveness, 
operational  suitability  and  operational  security  (USD  (AT&L)  2017,  104). 

Upon  successful  accomplishment  of  validation  of  the  capability,  Figure  5’s 
systems  engineering  “V”  concludes  with  the  system  proceeding  into  production 
and  fielding  to  the  warfighter  as  an  initial  operational  capability  (IOC)  and 
ultimately  full  operational  capability  (FOC)  (USD  (AT&L)  2017,  30). 

E.  DOD  PMO  TEST  PLANNING 

Within  a  DOD  PMO,  test  personnel  support  the  many  tasks  required.  As 
the  PMO  performs  its  initial  planning  and  as  potential  changes  are  assessed,  test 
personnel  support  the  development  of  different  COAs  in  support  of  schedule  and 
cost  activities.  During  this  COA  development  activity,  the  PMO  varies  the  number 
of  test  assets  available,  test  activities  to  be  performed,  and  the  test  time  that  is 
available.  This  COA  activity  is  completed  for  each  phase  of  the  acquisition  life 
cycle  where  test  assets  are  needed. 
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Multiple  schedule  COAs  are  generated  to  support  different  PMO  COAs. 
After  the  test  schedule  COAs  are  developed,  the  schedules  are  each  evaluated 
for  the  cost  associated  with  them.  Once  the  COA  is  decided  upon,  the  test 
schedule  that  supports  the  decided  upon  COA  becomes  the  system  test 
schedule.  As  the  program  proceeds  through  its  life  cycle,  there  are  many  COA 
activities.  The  schedule  is  subject  to  change  based  on  the  many  internal  and 
external  pressures,  some  of  which  are  shown  in  Figure  6. 


Figure  6.  Test  Schedule  Change  Pressures 


As  can  be  seen  from  Figure  6,  there  are  many  considerations,  limitations, 
constraints,  and  challenges  in  planning  and  executing  a  test  schedule.  In  addition 
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to  DT  planning,  PMO  test  schedulers  must  work  with  other  organizations  to 
determine  their  needs  for  operational  tests  and  live  fire  (LF)  tests,  and  plan 
appropriately  within  the  overarching  test  and  PMO  schedule  (USD  (AT&L)  2017, 
90).  The  LF  vehicles  become  vehicles  that  are  not  available  for  DT  testing,  and 
the  OT  window  confines  the  PMO  DT  test  schedule. 

Program  management  office  test  schedulers  must  work  with  test  sites  for 
test  site  availability,  recognizing  that  the  test  assets  may  not  be  delivered  as 
planned.  Test  schedulers  must  arrange  for  support  equipment  and  spares  to 
support  the  test  activities,  and  they  must  assure  that  test  and  supporting 
personnel  are  available  to  support  each  of  the  tests.  PMO  test  schedulers  must 
estimate  funding  needs  in  spite  of  the  potential  for  change  based  on  possible  test 
changes  due  to  late  delivery,  adverse  weather  conditions,  retests,  and 
maintenance  issues. 

Program  management  office  test  schedulers  must  address  requirements 
verification.  Test  schedulers  must  provide  the  PMO  with  confidence  that  the 
system  is  ready  to  move  into  OT  by  performing  test  events  of  an  OT  nature  such 
as  Reliability  Growth  Testing  (RGT).  PMO  test  schedulers  must  address  multiple 
COAs  and  must  plan  for  change  as  a  natural  course  of  doing  business  within  a 
DOD  PMO. 

Program  management  offices  require  a  dynamic  test  schedule  and  a 

dynamic  test  execution  process.  As  system  capabilities  are  changed,  these 

changes  may  result  in  system  requirements  changes,  which  change  test  event 

durations  and  test  events  needed  (USD  (AT&L)  2017,  92).  As  the  PMO  test 

personnel  work  with  other  test  organizations,  OT  and  LF  test  event  changes  may 

impact  the  test  timing.  As  funding  changes,  schedules  may  need  adjustment  to 

assure  execution.  As  the  test  assets  are  built,  issues  may  arise  in  production  that 

may  affect  the  delivery  schedule.  Test  assets  may  not  be  available  due  to 

maintenance  or  other  issues.  Test  sites  may  not  be  available  due  to  schedule 

changes  or  other  competing  activities  and  programs.  Personnel  needed  to 

support  test  events  may  not  be  available  as  planned  on  the  schedule,  and  test 
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events  may  require  certain  seasonal  weather,  which  may  require  shifts  in  other 
test  events  to  accommodate  timing  shifts.  Equipment  needed  for  a  test  event 
may  have  issues  with  availability.  These  examples  are  a  limited  set  of  the  many 
possible  test  schedule  change  pressures  as  indicated  in  Figure  6. 

F.  PROBLEM  STATEMENT 

This  research  explores  whether  optimization  can  aid  in  the  test  schedule 
development  process.  The  research  questions  addressed  by  this  thesis  are: 

•  Can  a  test  scheduling  model  automate  the  test  schedule 
development  process? 

•  Can  a  test  scheduling  model  optimize  the  PMO  test  scheduling 
activity  to  provide  multiple  optimized  test  schedule  options? 

•  Can  a  test  scheduling  model  determine  the  best  PMO  schedule  mix 
of  test  assets  and  test  facilities? 

These  research  questions  are  similar  to  the  overarching  critical 
operational  issues  (COIs)  that  are  asked  in  the  TEMP.  The  resulting  model  from 
this  research  will  be  evaluated  against  these  COIs  using  the  systems  engineering 
approach  identified  in  Figure  5. 

G.  ORGANIZATION  OF  THESIS 

This  thesis  is  organized  into  six  chapters,  with  model  input  and  output 
contained  in  Appendix  A,  and  3-D  monthly  schedule  conversions  contained  in 
Appendix  B. 

Chapter  I  introduces  the  thesis  by  providing  background  information  of  DOD 
processes  within  which  DOD  PMOs  and  their  test  planners  exist,  explaining  DOD 
PMO  test  planning  within  these  processes,  and  providing  the  problem  statement. 

Chapter  II  promotes  understanding  of  the  problem  by  establishing  the  test 
and  evaluation  (TE)  test  scheduling  process,  providing  the  model  problem 
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statement,  identifying  operational  needs,  exploring  model  alternatives,  and 
explaining  model  development  activities. 

Chapter  III  establishes  model  requirements  through  the  development  of  an 
integrated  computer  aided  manufacturing  definition  for  functional  modeling 
(IDEFO)  representation  of  model  requirements,  development  of  model  definitions, 
and  creation  of  detailed  requirements  based  on  the  IDEFO  representation. 
Requirements  are  organized  into  IDEFO  areas  of  inputs,  outputs,  controls  and 
constraints,  and  mechanisms,  resources,  and  tools. 

Chapter  IV  focuses  on  verification  of  the  model  against  the  model 
requirements  of  Chapter  III  and  continues  to  reflect  the  IDEFO  organization. 

Chapter  IV  uses  the  model  inputs  and  outputs  and  the  TE  3-D  schedule 
conversions  contained  in  Appendix  B  to  assess  the  model  against  the 
requirements. 

Chapter  V  validates  the  model  against  the  operational  needs  identified  in 
Chapter  II,  aggregating  the  model  3-D  schedules  showing  days  in  a  month  to  a 
higher  level  3-D  schedule  showing  months  in  a  year.  These  3-D  model  schedules 
are  compared  against  the  3-D  TE  planning  schedules. 

Chapter  VI  contains  the  conclusions  from  this  research  and  potential 
future  work. 
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II.  UNDERSTANDING  THE  PROBLEM 


A.  TE  TEST  SCHEDULING  PROCESS 

This  thesis  focuses  on  the  test  activities  within  the  Program  Manager 
Advanced  Amphibious  Assault  (PM  AAA)  vehicle  acquisition  programs.  The  PM 
AAA  PMO  supports  United  States  Marine  Corps  (USMC)  ship  to  shore  marine 
transport  systems. 

Within  the  PM  AAA  PMO,  PMO  test  personnel  develop  test  schedules  for 
each  COA,  per  Figure  7.  The  PM  AAA  test  schedule  development  process  shows 
the  activities  that  the  TE  test  planners  conduct  in  support  of  PM  AAA  schedule 
and  cost  development.  This  test  schedule  development  process  is  developed  in 
coordination  with  PM  AAA  PMO  test  personnel.  Although  this  process  documents 
the  specific  process  used  within  the  PM  AAA,  a  similar  process  would  be  used 
within  other  DOD  PMOs.  The  test  schedule  development  process  is  a  result  of 
this  thesis  research. 


19 


Role 


Test  and  Evaluation  Schedule  Development  Process 


Test 

Group 


’spec 


Syster  » 
Test 


Systei  » 
Test 


Test 


1.  Translate  Test 
Pspec  requirements 
into  Test  Events 


4.  Create  System 
Test  List 


5.  Update  System 
Test  List 


Test 


6.  Determine 
System  Test  Asset 
Schedule 


Vehicle 

Test 

cheduli 


2.  Identify  Number 
of  Test  Assets 
Available 


PM 

AAA 


3  Identify  Test  Time 
Available 


Figure  7.  PM  AAA  Test  Schedule  Development  Process 


Details  of  the  Figure  7  test  schedule  development  process  steps  are 
provided: 

Step  1.  Translate  System  Performance  Specification  Test 
Requirements  into  Test  Events.  The  first  step  in  the  test  schedule  development 
process  is  to  translate  the  system  performance  specification  test  requirements 
into  test  events.  During  this  step,  test  personnel  review  the  system  performance 
specification  test  requirements  and  identify  the  specific  test  events  that  need  to 
be  performed.  Test  events  are  identified  by  test  asset  type  since  there  may  be 
multiple  test  asset  variants.  Test  personnel  also  identify  the  number  of  days  of 
testing  needed  for  each  test  event,  how  many  test  assets  are  needed  to 
complete  each  test  event,  whether  the  test  requires  more  than  one  test  asset 
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simultaneously,  and  the  priority  of  the  test  event.  Step  1  is  performed 
simultaneously  with  Step  2  and  with  Step  3. 

Step  2.  Identify  Number  of  Test  Assets  Available.  The  second  step  in 
the  test  schedule  development  process  is  to  identify  the  number  of  test  assets 
available.  During  this  step,  the  PMO  identifies  and  provides  test  personnel  with 
the  total  number  of  test  assets  that  are  available  for  testing.  The  number  of  test 
assets  available  for  testing  excludes  the  LF  test  assets,  since  LF  tests  require 
dedicated  test  assets.  Test  assets  available  are  identified  for  each  test  asset 
type.  In  most  cases,  there  will  be  multiple  COAs  that  will  need  to  be  assessed, 
which  will  require  going  through  the  process  for  multiple  numbers  of  test  assets. 
Step  2  is  simultaneous  with  Step  1  and  with  Step  3. 

Step  3.  Identify  Test  Time  Available.  The  third  step  in  the  test  schedule 
development  process  is  for  the  PMO  to  identify  and  provide  the  test  time 
available  for  test  execution.  The  test  personnel  translate  the  overall  test  schedule 
time  into  the  total  number  of  time  periods  available.  This  involves  identifying  time 
periods  available  for  each  test  asset.  The  PMO  also  identifies  major  schedule 
events  and  decision  points,  which  will  constrain  the  test  schedule  through  test 
event  sequencing,  through  completion  times  required,  and  through  high,  medium 
and  low  priority  time  periods.  In  most  cases,  there  will  be  multiple  COAs  that  will 
need  to  be  assessed,  which  will  require  going  through  the  process  for  multiple 
test  times.  Step  3  is  simultaneous  with  Step  1  and  with  Step  2. 

Step  4:  Create  System  Test  List.  The  first  step  is  followed  by  a  review  of 
prior  system  test  lists  to  identify  a  system  test  list  that  is  close  to  what  the  new 
system  test  list  needs  to  contain.  The  test  personnel  then  use  the  selected  prior 
system  test  list  as  a  starting  point  and  update  it  to  contain  the  information  needed 
in  the  new  system  test  list.  The  test  personnel  review  the  system  test  list  for 
changes  based  on  the  activities  for  step  1  and  step  3.  Depending  on  the  test 
events  needed,  there  may  be  deletions,  updates  and  additions. 
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Step  5.  Update  Test  List.  During  this  step,  test  personnel  identify  test 
agencies  and  test  facilities  that  can  perform  each  of  the  test  events  and  identify 
the  preferential  order  that  they  would  chose  the  test  agency  for  each  test  event. 
Test  personnel  identify  all  test  event  test  sequencing  relationships,  predecessor 
test  events,  and  test  event  priorities  (high  medium  and  low).  These  process 
activities  are  performed  for  each  test  asset  type.  Test  personnel  place  test  events 
into  test  functional  groups,  called  critical  technical  parameter  (CTP)  areas,  which 
will  be  reflected  on  the  higher  level  published  test  schedule. 

Step  6.  Determine  System  Test  Asset  Schedule.  The  last  step  in  the 
test  schedule  development  process  is  to  determine  the  system  test  asset 
schedule  using  inputs  from  the  previous  steps.  The  intent  of  this  step  is  to 
determine  the  best  system  test  asset  schedule  to  minimize  facility  movement  of 
each  test  asset  and  to  minimize  the  completion  time  of  all  test  events  while 
maintaining  the  constraints  of  prioritization,  sequencing,  and  completion  times. 
Multiple  schedules  are  produced  during  this  step  for  each  set  of  test  asset 
numbers  and  test  time  periods,  resulting  in  a  best  case  minimum  schedule,  a 
worst  case  maximum  test  schedule  and  the  most  likely  test  schedule. 

Test  scheduling  is  currently  performed  using  Microsoft  (MS)  Excel  through 
a  series  of  meetings  with  multiple  PMO  test  personnel.  PMO  test  personnel  use 
heuristics,  based  on  their  knowledge  and  experience,  understanding  what  test 
events  are  predecessor  and  successor  tasks,  the  relative  priority  of  the  test 
events,  which  test  facilities  can  perform  the  test  events,  which  test  facilities  are 
preferred  to  perform  the  test  events,  how  many  test  assets  are  needed  for  the 
test  events,  and  simultaneous  testing  test  facility  bandwidth.  Initial  test 
scheduling  activity  takes  weeks  to  complete. 

B.  MODEL  PROBLEM  STATEMENT 

Developed  in  coordination  with  this  research,  the  following  statement  is 
the  quoted  problem  statement  from  the  model  developer’s  thesis  and  consists  of 
multiple  paragraphs: 
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There  are  several  variants  of  test  asset  (e.g.,  pieces  of  a  type  of 
equipment  to  be  tested)  that  need  to  be  subjected  to  a  set  of  test 
events  conducted  at  a  number  of  test  venues  (i.e.,  test  facilities). 

Each  test  event  may  apply  to  some  subset  of  test  asset  variants, 
and  may  be  performed  by  any  suitably  equipped  test  venue. 

The  planning  horizon  consists  of  discrete,  ordered  time  periods 
(say,  days).  Each  test  asset  is  to  be  initially  delivered  to  a  test 
venue  at  the  start  of  a  given  scheduled  time  period,  but  may  be 
subsequently  moved  among  other  venues.  Completing  each  test 
event  requires  visiting  a  test  venue  for  some  given  number  of 
contiguous  time  periods.  Moving  a  test  asset  from  one  test  venue  to 
another  venue,  and  inspecting  it  on  receipt,  requires  a  given 
number  of  contiguous  time  periods.  A  test  asset  located  at  a  test 
venue  may  be  held  back  for  other  activities,  and  thus  be 
unavailable  for  testing  during  some  time  periods.  A  test  asset  can 
only  undergo  a  single  test  event  during  any  time  period,  and  each 
test  event  will  be  conducted  at  most  once  during  the  planning 
horizon. 

Each  test  event  has  a  priority  (an  ordered  attribute),  and  all  higher- 
priority  test  events  should  be  started  before  any  lower-priority  ones 
are  started,  and  completed  before  a  priority-specific  deadline  day. 
Lowest-priority  tests  can  be  completed  at  convenience,  including 
past  the  end  of  the  planning  horizon  (i.e.,  these  are  optional  tests). 

Some  tests  have  precedence  over  others,  and  are  required  to  be 
completed  before  the  others  are  started,  independent  of  their 
priority.  All  test  events  of  or  above  a  given  priority  threshold  must 
be  completed,  and  the  objective  is  to  minimize  completion  time  of 
the  last  of  these  tests. 

Each  test  venue  has  a  limit  on  the  number  of  test  assets  it  can 
accommodate  at  any  time,  but  there  is  no  limit  on  test  venue 
capacity  to  perform  simultaneous  tests.  (Edwards  2015,  15) 

This  problem  can  be  represented  in  a  three-dimensional  space,  with  the 
coordinates  of  time,  test  facility,  and  test  event,  with  the  test  asset  variants 
moving  through  time,  as  shown  in  Figure  8.  Solutions  are  constrained  by  which 
test  events  are  needed  by  variant,  by  what  test  facilities  can  support  the  test 
events,  by  relationships  between  the  test  events,  by  test  facilities  that  can 
complete  the  test  events,  by  test  event  priorities,  by  priority  deadlines,  and  by 
overall  schedule  completion  times.  Therefore,  there  is  a  subset  of  the  entire 
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space  of  available  solutions  that  will  meet  the  constraints,  and  there  is  an 
optimized  solution  or  set  of  solutions  to  minimize  the  total  test  schedule  time  and 
to  minimize  the  test  asset  movement  between  facilities. 


Figure  8.  Test  Schedule  Problem  3-D  Representation 


Figure  8  is  an  illustrative  example  of  an  optimal  solution  for  four  test 
assets,  A-1,  A-2,  B-1  and  B-2.  The  test  assets  designated  as  “A-x”  are  one 
variant  type,  whereas  the  test  assets  designated  as  “B-y”  are  a  different  variant 
type.  Each  of  these  test  assets  move  from  point  zero  through  the  solution  space 
over  time  through  the  points,  showing  the  chosen  test  event  at  a  chosen  test 
facility  at  a  point  in  time,  until  the  entire  set  of  test  events  are  completed.  For 
example,  test  asset  A-1  (blue)  begins  time  period  one  starting  at  test  facility  2, 
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which  conducts  test  event  1 .  A-1  then  moves  at  time  period  2  to  test  facility  5  to 
complete  test  event  4.  At  time  period  3,  A-1  moves  to  test  facility  4  to  complete 
test  event  3.  Finally,  A-1  moves  to  test  event  1  at  test  facility  2  at  time  period  4. 
Not  every  point  of  the  three-dimensional  space  is  available  to  every  test  asset. 
There  is  a  set  of  points  available  to  each  variant  set  (a  set  for  “A-x”s,  and  a  set 
for  “B-y”s). 


C.  MODEL  OPERATIONAL  NEEDS 

As  illustrated  in  Figure  5,  the  first  step  in  the  systems  engineering  “V”  is 
the  development  of  model  operational  needs,  which  are  given  in  Table  1 . 


Table  1.  Model  Operational  Needs 


Need  # 

Title 

Tier 

Operational  Needs 

N1 

Low  Priority  Test 
Events  (0) 

A  PA 

The  model  low  priority  test  events  shall 
be  allowed  to  go  beyond  the  test  period. 

N2 

Multiple  Assets 
(T=0) 

KSA 

The  model  shall  allow  multiple  test  assets 
to  be  tested  simultaneously. 

N3 

Time  Period  Asset 
Availability  (0) 

A  PA 

The  model  shall  have  a  time  period  test 
asset  availability  constraint. 

N4 

Venue  Distance 
(T=0) 

KSA 

The  model  shall  minimize  test  movement 
between  test  venues. 

N5 

Event  Priority 
Placement  (T=0) 

KSA 

The  model  shall  place  test  events  based 
on  priority. 

N6 

Multiple  Venues 
(T=0) 

A  PA 

The  model  shall  allow  multiple  test 
venues  to  be  used  at  the  same  time. 

N7 

Venue  Choice  by 
Event  (T=0) 

A  PA 

The  model  shall  allow  test  venue  choice 
by  test  event. 

N8 

Multiple  Events 
(T=0) 

A  PA 

The  model  shall  allow  multiple  test  events 
to  occur  at  the  same  time, 

N9 

Precedence 
Relationship  (T=0) 

A  PA 

The  model  shall  have  test  event 
predecessor  and  successor  relationship 
constraints. 

N10 

Deadline  for  each 
Priority  (T=0) 

A  PA 

The  model  shall  place  test  assets  based 
on  a  deadline  for  each  priority  type. 

Nil 

Schedule  Test 

Events  (T=0) 

KPP 

The  model  shall  minimize  the  time  period 
used  for  test  events. 

N12 

MS  Excel  Input 
(T=0) 

A  PA 

The  model  shall  allow  TE  personnel  to 
input  information  in  MS  Excel. 
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Need  # 

Title 

Tier 

Operational  Needs 

N13 

MS  Excel  Output 
(T=0) 

KPP 

The  model  shall  output  schedules  in  a 
MS  Excel  format  showing  each  vehicle, 
using  colors  for  test  site,  and  using 
blocks  that  aggregate  the  information  into 
the  month. 

N14 

Model  in  MS  Excel 
(0) 

APA 

The  model  shall  use  MS  Excel  only.  This 
means  that  a  specialized  tool  will  not  be 
used  for  the  model. 

These  model  operational  needs  are  prioritized  by  tier  and,  as  discussed  in 
CJCSI  3170.01  (2015b,  D-61),  are  identified  as  a  KPP,  key  system  attribute 
(KSA)  or  additional  performance  attribute  (APA).  Key  performance  parameters 
are  the  highest  level  capability  and  require  involvement  at  the  highest  level  to 
change  them.  If  KPPs  are  not  met,  the  program  itself  is  in  jeopardy,  since  these 
capabilities  are  the  basis  for  the  program’s  existence.  Key  system  attributes  are 
the  second  most  important  tier,  but  the  milestone  decision  authority  (MDA)  has 
the  authority  to  change  them.  Additional  performance  attributes  are  the  third  level 
of  priority  and  can  be  changed  by  the  capability  drafter.  Objective  capabilities  are 
the  fourth  level  of  priority,  as  they  are  desired  capabilities. 

D.  RELATED  PROBLEMS  AND  MODELS 

There  are  many  models  that  are  available  to  assess  a  schedule  once  it 
has  been  developed.  Providing  a  schedule  model  based  on  inputs  and 
constraints  is  a  much  more  difficult  activity. 

Research  is  performed  relative  to  other  schedule  optimization  problems, 
models  and  methods  used  to  solve  the  problems,  and  the  applicability  of  the 
problems  to  the  schedule  optimization  capabilities  needed.  As  illustrated  in 
Figure  5  under  operational  needs,  the  first  step  in  the  systems  engineering  “V” 
includes  performing  market  surveys.  As  these  models  have  already  been 
developed,  the  question  is  whether  they  could  be  modified  to  address  the 


26 


operational  needs  of  PMO  test  scheduling  using  the  operational  needs 
expressed  in  Table  1 . 

The  first  optimization  decision  making  model  explored  is  “Using 
Optimization  to  Improve  NASA  Extravehicular  Activity  Planning,”  hereafter 
referred  to  in  this  thesis  as  the  extravehicular  activity  (EVA)  model.  This  model 
prioritizes  tasks  for  one  to  two  crew  and  tasks  with  different  durations.  The  EVA 
model  addressed  tasks  with  different  priorities,  tasks  with  predecessor 
relationships,  tasks  at  different  locations,  and  tasks  with  time  period  availability 
constraints  over  a  given  time  period.  Tasks  also  have  information  about  whether 
they  are  mandatory,  the  number  of  crew  required,  whether  they  have  a 
contamination  risk,  and  if  they  have  to  be  completed  if  started,  referred  to  as 
“bingo  time”  (Felker  2012,  36). 

The  second  optimization  decision  making  model  explored  is  “Pacific  Fleet 
Submarine  Tender  Optimization,”  hereafter  referred  to  in  this  thesis  as  “SUB,” 
The  SUB  model  prioritizes  maintenance  tasks  for  workers  at  different  locations 
over  a  given  time  period  while  minimizing  travel.  Tasks  also  have  information 
about  “whether  or  not  tender  presence  is  required;  the  estimated  total  number  of 
worker-days  required;  the  beta_max  number  of  workers  that  can  simultaneously 
work  on  each  task;  the  types  of  maintenance  workers  that  can  perform  the  task; 
and  task  due  dates”  (Pickett  201 3,  i). 

The  third  optimization  decision  making  model  explored  is  “An  optimization 
of  The  Basic  School  Military  Occupational  Skill  Assignment  Process,”  hereafter 
referred  to  in  this  thesis  as  military  operational  skills  (MOS)  model.  The  MOS 
model  assigns  MOSs  to  Lieutenants  based  on  class  and  leadership  standing, 
available  MOSs  and  preferences  (Boersma  2003,  49). 

The  fourth  optimization  decision  making  model  explored  is  “Optimizing 
Marine  Security  Guard  Assignments,”  hereafter  referred  to  in  this  thesis  as  the 
marine  security  guards  (MSG)  model.”  The  MSG  model  “assign[s]  [MSGs]  to 
billets  and  to  balance  MSG  experience  levels  across  all  detachments”  (Enoka 
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2011,  15).  MSG  assignments  are  based  on  detachment  attributes  and  billet 
attributes  of  experience  level,  rank,  restrictions,  region,  gender,  tier,  preference, 
priority,  and  qualification  (Enoka  201 1,21). 

The  fifth  optimization  decision  making  model  explored  is  “Optimizing  Ship- 
to-Shore  Movement  for  Hospital  Ship  Humanitarian  Assistance  Operations,” 
hereafter  referred  to  in  this  thesis  as  the  Hospital  model.  The  hospital  model 
“determine(s)  transportation  asset...  routing  and  loading  to  effect  the  movement 
of  personnel  and  patients  between  ship  and  ashore  mission  site”  (Ward  2008,  5). 
This  problem  addresses  daily  routing  of  multiple  assets  and  multiple  asset  types 
through  different  nodes,  with  priority  sites.  This  model  is  constrained  by  how 
many  assets  can  be  at  each  node  simultaneously  and  when  and  where 
personnel  need  to  be  picked  up  and  delivered  (Ward  2008,  29). 

The  operational  needs  identified  in  Table  1  are  assessed  for  each  model. 
Table  2  shows  the  operational  needs  and  what  each  of  the  models  assessed 
addresses  against  these  needs.  A  “Yes”  indicates  that  the  model  addresses  the 
operational  need.  A  “No”  indicates  that  the  model  does  not  address  the  identified 
operational  need. 

In  addition  to  the  operational  need  assessment  for  each  model,  the 
problem  type  and  objective  function  are  identified  in  Table  2  for  information  and 
comparison. 
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Table  2.  Model  Alternatives  Explored 


Model 

Capability 

EVA  Model 

SUB  Model 

MOS  Model 

MSG  Model 

Hospital 

Model 

Low  Priority 
Test  Events 
(N1) 

No 

No 

No 

No 

No 

Multiple 

Assets  (N2) 

KSA 

Yes,  Two 
people 

Yes.  Multiple 
people 

Yes.  Multiple 
People 

Yes.  Multiple 
People 

Yes 

Time  Period 
Asset 
Availability 
(N3) 

Yes 

Yes 

No 

No 

No 

Venue 

Distance  (N4) 
KSA 

Yes.  Location 
Proximity 

Yes.  Different 
locations 

No 

No 

Yes 

Event  Priority 

Placement 

(N5) 

KSA 

Yes,  Priority 
tasks 

Yes.  Priority 
tasks 

Yes.  Class 
standing  takes 
MOS  priority 

Yes.  Billet 

priority 

placement 

Yes 

Multiple 

Venues (N6) 

Yes,  Multiple 
Locations  on 
the  space 
station 

Yes 

Yes.  Multiple 
MOS  Types 

Yes 

Yes 

Venue  Choice 
by  Event  (N7) 

Yes,  Chose 
location  for 
task 

No 

Yes.  MOS 
preferences 

Yes.  MSG 
preferences 

No 

Multiple 

Events  (N8) 

Yes,  Multiple 
tasks 

Yes.  Multiple 
tasks 

No 

No 

Yes 

Precedence 

Relationships 

(N9) 

Yes 

Yes 

No 

No 

No 

Deadline  for 
each  Priority 
(N10) 

No 

Yes.  Task 

completion 

dates 

No 

No 

No 

Schedule  Test 

Yes. 

No.  Schedule 

No.  Assign 

No.  Assign 

No.  Schedule 

Events  (Nil) 

Schedule 

SUB 

Personnel  to 

MSGs  to 

Transportation 

KPP 

EVA  Tasks 

Maintenance 

Tasks 

MOS 

Embassy 

Posts 

Assets 

MS  Excel 

Input  (N12) 

Yes 

Yes 

Yes 

Yes 

Unknown 

MS  Excel 
Output  (N1 3) 
KPP 

No.  GAMS 

No.  GAMS 

Yes 

Yes 

Unknown 

Model  in  MS 
Excel  (N14) 

No.  Uses 
GAMS 

No.  Uses 

GAMS 

No.  Uses 
solver.com 

Yes 

No.  Xpress-MP 

Problem  Type 

Mixed  Integer 
Linear 

Mixed  Integer 
Linear 

Integer  Linear 

Integer  Linear 
Multi- 

commodity 

Network 

Flow 

Multi-objective 
Mixed  Integer 
Linear 

Objective 

Maximize 

Minimize 

Minimize  MOS 

Minimize 

Minimize 

Function 

Priority  Tasks 

completion  time 
and  late  tasks 

not  meeting  Lt 
choice  list 
based  on  class 
standing 

MSG  not 
assigned  to 
Billets  without 
needed 
experience 

Transportation 

Schedule 
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While  there  is  much  similarity  between  our  test  schedule  problem  and  the 
EVA  problem,  there  is  a  significant  difference.  Our  problem  is  to  complete  all  of 
the  tasks  within  a  certain  timeframe,  with  different  completion  times  based  on 
priority  and  with  some  tasks  allowed  to  go  beyond  the  timeframe.  The  EVA 
problem  is  to  decide  which  tasks  to  perform  based  on  priority  within  a  given 
timeframe.  Since  this  problem  has  different  constraints  and  due  to  the  additional 
complexity  of  our  problem,  the  EVA  problem  solution  is  not  used  as  a  starting 
point  for  our  model. 

There  is  considerable  similarity  between  our  test  schedule  problem  and 
the  SUB  problem.  The  differences  are  that  we  have  multiple  venues  to  choose 
from,  and  our  specific  problem  and  constraints  are  different.  Our  problem  is  to 
minimize  the  completion  time  of  all  tasks.  Although  similar  in  approach,  the  SUB 
problem  solution  is  not  used  as  a  starting  point  for  our  model  due  to  these 
differences. 

The  MOS  problem  is  very  simplistic  in  comparison  with  our  schedule 
problem,  and  does  not  have  much  similarity  with  our  problem.  Therefore,  this 
problem  solution  is  not  used  as  a  starting  point  for  our  model. 

Like  the  MOS  problem,  the  MSG  problem  is  very  simplistic  in  comparison 
with  our  schedule  problem,  and  does  not  have  much  similarity  with  our  problem. 
Thus,  the  MSG  problem  solution  is  not  used  as  a  starting  point  for  our  model. 

There  is  some  similarity  between  our  test  schedule  problem  and  the 
Hospital  problem.  We  have  multiple  venues  to  choose  from,  we  have 
precedence,  and  we  have  priority  completion  times,  whereas  the  Hospital 
problem  does  not.  Also,  our  specific  problem  and  constraints  are  different.  Due  to 
these  many  differences,  the  Hospital  problem  solution  is  not  used  as  a  starting 
point  for  our  model. 

As  part  of  model  review,  there  are  two  other  optimization  decision  making 
models  that  are  also  explored.  These  two  models  incorporate  cost  into  the 
model. 
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The  first  cost  optimization  decision  making  model  explored  is  “The  United 
States  Army’s  Multi-period  Optimal  Readiness  Allocation  Model,”  hereafter 
referred  to  in  this  thesis  as  “EOT.”  This  model  “maximize(s)  unit  readiness  by 
determining  equipment  (re)distribution  plan  for  every  year  of  the  POM,” 
“measures  readiness  as  a  weighted  sum  of  unit  S-ratings  and  LIN  S-ratings 
across  all  units,”  and  “expands  it  over  the  years  of  the  planning  horizon”  (Parsons 
2011,  18).  In  the  EOT  model,  “Unit  S-ratings  weights  are  assigned  based  on  the 
priority  of  the  unit.  A  similar  construct  is  used  to  assign  each  unit’s  LIN  S-rating 
weights”  (Parsons  201 1 ,  18). 

The  second  cost  optimization  decision  making  model  explored  is  “Cost- 
Constrained  Project  Scheduling  with  Task  Durations  and  Costs  that  may 
Increase  over  Time  Demonstrated  with  the  U.S.  Army  Future  Combat  System,” 
hereafter  referred  to  in  this  thesis  as  “Sch  Cost.”  The  Sch  Cost  model  explores 
“feasible  task  schedules,  selecting  those  that  minimize  the  length  of  the  project 
critical  path  while  observing  annual  and  project  constraints”  (Grose  2015,  19). 

While  there  is  cost  associated  with  test  schedules,  in  PMAAA,  costs  are 
determined  by  the  TE  planners  after  the  schedules  are  built.  Therefore,  these 
models  are  not  considered  as  potential  solutions  to  the  problem  at  hand.  It  is 
noted,  however,  that  if  the  model  developed  by  this  thesis  is  matured,  that  the 
cost  of  each  schedule  could  potentially  be  incorporated  into  the  output  of  the 
model  and  could  be  included  in  the  decision  process  of  the  model  itself. 

Based  on  this  model  AoA  review  against  the  operational  needs,  a  model  is 
determined  to  be  needed  to  address  the  problem  statement.  Additionally,  based 
on  the  complexity  of  the  problem,  it  is  expected  the  Microsoft  Excel,  even  with 
solver  implemented,  may  not  be  able  to  address  this  complex  problem. 
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III.  MODEL  REQUIREMENTS  AND  MODEL  DEVELOPMENT 


A.  IDEFO  MODEL  REPRESENTATION 

As  illustrated  in  Figure  5,  the  next  step  in  the  systems  engineering  “V”  is 
the  decomposition  of  the  operations  needs  into  system  requirements.  The 
requirements  and  verification  criteria  are  developed  from  the  operational  needs 
and  the  problem  statement. 

Developing  system  requirements  results  in  a  large  number  of 
requirements.  Thus,  having  a  method  to  organize  the  requirements  is  helpful. 
IDEFO  is  used  in  this  research  to  perform  this  requirement  organizational 
function.  The  majority  of  system  requirements  are  functional  requirements,  which 
means  that  the  system  requirements  define  the  functions  that  the  system  needs 
to  perform.  Functional  analysis  and  synthesis  is  the  process  used  in  system 
engineering  to  generate  the  system  functional  requirements. 

IDEFO  is  a  method  commonly  used  in  systems  engineering  when 
performing  functional  analysis  and  synthesis,  and  is  defined  by  DAU  as 

a  model  that  consists  of  a  hierarchical  referenced  to  each  other. 

The  two  primary  modeling  components  are:  functions  (represented 
on  a  diagram  by  boxes),  and  data  and  objects  that  interrelate  those 
functions  (represented  by  arrows).  The  position  at  which  the  arrow 
attaches  to  a  box  conveys  the  specific  role  of  the  interface.  The 
controls  enter  the  top  of  the  box.  The  inputs,  the  data  or  objects 
acted  upon  by  the  operation,  enter  the  box  from  the  left.  The 
outputs  of  the  operation  leave  the  right-hand  side  of  the  box. 
Mechanism  arrows  that  provide  supporting  means  for  performing 
the  function  join  (point  up  to)  the  bottom  of  the  box.  (DAU  2001 , 51 ) 

Using  the  IDEFO  method  to  structure  the  functional  requirements  of  the 
model,  Figure  9  is  developed  for  the  model  requirements.  Figure  9  shows  the 
inputs,  the  outputs,  the  controls  and  constraints,  and  the  mechanisms,  resources, 
and  tools  of  the  model.  Figure  9  provides  short  statements  that  are  the  titles  of 
the  detailed  requirements  in  the  model  requirements  Section. 
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Controls  and  Constraints 


Mechanisms,  Resources,  Tools 


Figure  9.  IDEFO  Model  Representation 
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B.  MODEL  DEFINITIONS 


Part  of  developing  model  requirements  is  having  common  definitions  in 
order  to  avoid  misunderstanding.  Definitions  provided  here  are  adapted  from  the 
problem  statement,  which  was  developed  in  coordination  with  Shane  A.  Edwards 
(Edwards  2015,  15),  and  are  given  in  alphabetical  order: 

Test  Asset.  The  item  that  is  being  tested.  Test  assets  may  have  test 
periods  during  which  they  are  unavailable.  There  may  be  multiple  test  assets  of 
each  test  asset  type. 

Test  Asset  Type.  A  variant,  or  type,  of  test  asset,  that  is  sufficiently 
different  from  another  test  asset  such  that  it  requires  separate  testing  for  all  or 
some  of  the  test  events  planned  for  the  other  test  asset  type(s).  There  may  be 
one  or  more  test  asset  types  that  go  through  test  events. 

Test  Event.  An  event  where  specific  tests  are  performed  on  a  test  asset.  A 
test  event  may  apply  to  one  or  more  test  asset  type(s),  may  require  one  or  more  test 
assets,  and  may  require  one  or  more  test  asset  types. 

Test  Event  Precedence.  A  test  event  relationship  that  requires  that  a 
specific  test  event  be  completed  before  another  test  event  can  begin.  Test  event 
precedence  may  be  test  asset  type  specific. 

Test  Event  Priority.  The  priority  state  of  the  test  event.  Values  of  the  test 
event  priority  are  high,  medium,  and  low. 

Test  Event  Priority  Deadline.  The  test  event  priority  deadline  is  defined 
as  the  last  time  period  that  a  test  event  can  be  completed.  High  and  medium 
priority  test  events  have  test  event  priority  deadlines.  Low  priority  test  events  do 
not  have  a  deadline  and  can  be  completed  after  the  test  period. 

Test  Period.  The  time  period  available  to  perform  a  test  event.  It  consists 
of  one  or  more  test  periods.  When  moving  test  assets  between  test  venues,  test 
periods  are  needed  for  movement  to,  and  for  inspection  at,  the  receiving  test 
venue. 
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Test  Venue.  A  facility  where  a  test  asset  can  be  tested.  There  may  be 
multiple  test  venues  that  can  complete  a  specific  test  event.  Not  all  test  venues 
can  complete  a  given  test  event.  There  may  be  time  periods  where  a  test  venue 
is  not  available.  Test  venues  are  limited  in  the  number  of  test  assets  that  they 
can  accommodate.  Test  venues  may  perform  simultaneous  test  events  on  test 
assets  at  the  test  venue. 

Time  Period.  A  duration  of  time,  such  as  days.  Test  events  can  be 
allocated  to  test  venues  during  the  time  period. 

C.  MODEL  REQUIREMENTS 

The  test  and  evaluation  resource  constrained  scheduling  problem  (RSCSP) 
model  requirements  and  verification  criteria  are  provided  in  this  section.  These 
requirements  are  broken  down  in  detail  in  Tables  3  through  6.  The  requirements 
found  in  these  tables  correspond  with  Figure  9.  Table  3  corresponds  with  IDEFO 
model  input  requirements  (requirement  “Para  #”  of  1.x).  Table  4  corresponds  with 
IDEFO  model  control  and  constraint  requirements  (requirement  “Para  #”  of  2.x). 
Table  5  corresponds  with  IDEFO  model  mechanisms,  resources  and  tools 
requirements  (requirement  “Para  #”  of  3.x).  Table  6  corresponds  with  IDEFO  model 
output  requirements  (requirement  “Para  #”  of  4.x).  The  blocks  given  in  the  Figure  9 
IDEFO  model  correspond  with  the  titles  of  the  requirements  that  are  provided  in 
Tables  3  through  6.  Requirement  and  verification  details  are  given  in  the 
“Requirement  and  Verification  Criteria”  column  in  Tables  3  through  6.  The  “Need  # 
Relationship”  column  in  Tables  3  through  6  provides  a  relationship  between  the 
model  requirement  and  operational  need  capabilities  of  Table  1 . 

For  simplicity,  requirements  and  verification  criteria  have  been  combined 
into  a  single  requirement  and  verification  criteria.  Interpretation  of  these  tables 
means  that  a  requirement  consists  of  the  first  table  row  combined  with  each  row 
below  the  first  row.  For  example,  the  1 .1  requirement  is  pulled  from  Table  3  rows 
1 .0  (The  TE  RSCSP  model  shall  accept  the  following  input,  which  is  verified  by 
reviewing  the  model  input  file)  and  1.1  (Test  Period  (T=0)).  Therefore,  the  1.1 
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requirement  is  “The  TE  RSCSP  model  shall  accept  the  following  input,  which  is 
verified  by  reviewing  the  model  input  file:  Test  Period  (T=0).”  The  requirements 
are  verified  in  Chapter  IV  and  validated  in  Chapter  V  using  the  input  files,  the 
model  generated  output  file,  and  the  manually  converted  3-D  TE  schedules. 

Table  3  provides  the  TE  RSCSP  model  1.x  requirements,  the  model 
verification  criteria,  and  the  requirement  relationship  to  the  operational  need  for 
model  input  requirements,  using  the  “Inputs”  boxes  from  Figure  9. 


Table  3.  Model  Input  Requirements 


Para 

# 

Requirement  and  Verification  Criteria 

Need  # 
Relationship 

1.0 

The  TE  RSCSP  model  shall  accept  the  following  input, 
which  is  verified  by  reviewing  the  model  input  file: 

N/A 

1.1 

Test  Period  (T=0) 

Nil 

1.2 

Test  Events  (T=0) 

N5 

1.3 

Test  Asset  Type  (T=0) 

N2 

1.4 

Test  Venues  (T=0) 

N7 

1.5 

Test  Event  Priorities  (T=0) 

N5 

1.6 

Test  Venue  Movement  Test  Periods  (T=0) 

N4 

1.7 

Test  Venue  Test  Asset  Capacity  (T=0) 

N6 

1.8 

Test  Event  Priority  Relationships  (T=0) 

N5 

1.9 

Test  Event  Test  Venue  Relationships  (T=0) 

N7 

1.10 

Test  Event  Precedence  Relationships  (T=0) 

N9 

1.11 

Test  Event  Test  Period  Durations  (T=0) 

Nil 

1.12 

Number  of  Test  Assets  Needed  for  Test  Events  (T=0) 

N2 

1.13 

Test  Asset  Type  Test  Event  Relationships  (T=0) 

N2 

1.14 

Test  Asset  Test  Venue  Starting  Location  (T=0) 

N7 

1.15 

Test  Asset  Unavailability  (0) 

N3 

1.16 

Test  Venue  Unavailability  (0) 

N6 

1.17 

Test  Event  Priority  Deadlines  (T=0) 

N10 

1.18 

Priority  Deadline  Penalties  (T=0) 

N10 

Table  4  provides  the  TE  RSCSP  model  2.x  requirements,  the  model 
verification  criteria,  and  the  requirement  relationship  to  the  operational  need  for 
model  control  and  constraint  requirements,  using  the  “Control  and  Constraint” 
boxes  from  Figure  9. 
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Table  4.  Model  Control  and  Constraint  Requirements 


Para 

# 

Requirement  and  Verification 

Need  # 
Relationship 

2.0 

Controls  and  Constraints  (T=0).  The  TE  RSCSP  model 
shall  accept  and  use  model  controls  and  constraints, 
which  are  verified  by  reviewing  the  model  files  and  3-D 
conversions: 

N/A 

2.1 

Test  Period  (T=0) 

Nil 

2.1.1 

One  Test  Event  on  Test  Asset  during  Time  Period  (T=0) 

N5 

2.1.2 

Place  all  Test  Events  in  Time  Period  (T=0) 

N5 

2.1.3 

Test  Event  Test  Period  Durations  (T=0) 

Nil 

2.2 

Test  Events  (T=0) 

N5 

2.3 

Test  Asset  Type  (0) 

N2 

2.4 

Test  Venues  (T=0) 

N7 

2.5 

Test  Event  Priority 

N/A 

2.5.1 

Test  Event  Priority  Relationships  (T=0) 

N5 

2.5.2 

High  to  Medium  Priority  Relationship  (T=0) 

N5 

2.5.3 

Medium  to  Low  Priority  Relationship  (T=0) 

N5 

2.5.4 

High  to  Low  Priority  Relationships  (T=0) 

N5 

2.6 

Test  Venue  Movement  Test  Periods  (T=0) 

N4 

2.6.1 

Add  Time  Period  from  Test  Venue  Movement  (T=0) 

N4 

2.6.2 

Schedule  Test  Event  on  Available  Test  Venues  (T=0) 

N7 

2.7 

Test  Venue  Test  Asset  Capacity  (T=0) 

N7 

2.8 

Test  Asset  Test  Venue  Starting  Location  (T=0) 

N7 

2.9 

Test  Event  Test  Venue  Relationships  (T=0) 

N7 

2.10 

Test  Event  Precedence  Relationships  (T=0) 

N9 

2.10.1 

Test  Event  Predecessor  First  (T=0) 

N9 

2.10.2 

Test  Event  Successor  Second  (T=0) 

N9 

2.11 

Number  of  Test  Assets  Needed  for  Test  Events  (T=0) 

N8 

2.12 

Low  Priority  Schedule  Relationship  (0) 

N1 

2.13 

Test  Asset  Unavailability  (0) 

N3 

2.14 

Test  Venue  Unavailability  (0) 

N7 

2.15 

Test  Event  Priority  Deadlines  (T=0) 

N10 

2.15.1 

High  Priority  to  Deadline  Relationship  (T=0) 

N10 

2.15.2 

Medium  Priority  to  Deadline  Relationship  (T=0) 

N5 

2.15.3 

High  Priority  to  Test  Period  Relationship  (T=0) 

N5 

2.15.4 

Medium  Priority  to  Test  Period  Relationship  (T=0) 

N5 

Table  5  provides  the  TE  RSCSP  model  3.x  requirements,  the  model 
verification  criteria,  and  the  requirement  relationship  to  the  operational  need  for 
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model  mechanisms,  resources  and  tools  requirements,  using  the  “Mechanisms, 
Resources  and  Tools”  boxes  from  Figure  9. 


Table  5.  Model  Mechanisms,  Resources  and  Tools  Requirements 


Para 

# 

Para  Title 

Requirement  and  Verification 
Criteria 

Need  #  1 

Relationship  1 

3.0 

Mechanisms, 
Resources  and  Tools 

The  TE  RSCSP  model  tool 
requirements  shall  be  verified  by 
reviewing  the  model  files: 

N/A 

3.1 

MS  Excel  Input  (T=0) 

The  format  of  the  TE  RSCSP 
model  input  files  shall  be  MS 

Excel. 

N12 

3.2 

MS  Excel  Model 
(Objective) 

The  TE  RSCSP  model  shall  use 
MS  Excel  for  calculation,  output 
and  display  similar  to  current  TE 
output. 

N14 

Table  6  provides  the 

TE  RSCSP  model  4.x  requirements,  the  model 

verification  criteria,  and  the  requirement  relationship  to  the  operational  need  for 
model  output  requirements,  using  the  “Outputs”  boxes  from  Figure  9. 

Table  6. 

Model  Output  Requirements 

Para 

# 

Para  Title 

Requirement  and  Verification 
Criteria 

Need  # 
Relationship 

4.0 

Model  Outputs 

The  TE  RSCSP  model  output 
requirements  shall  be  verified  by 
reviewing  the  model  output  files: 

N13 

4.1 

MS  Excel  Model 
Output  (T=0) 

The  TE  RSCSP  shall  use  MS 

Excel  for  the  model  output. 

N13 

4.2 

Model  Format  (T=0) 

The  TE  RSCSP  Model  shall  be  in 
the  same  format  as  the  TE 
manual  method: 

N13 

4.2.1 

Model  Format  - 
Months  (T=0) 

Months 

N13 

4.2.2 

Model  Format  -  Test 
Periods  (T=0) 

Test  periods 

N13 

4.2.3 

Model  Form  -  Test 
Assets  (T=0) 

Test  assets 

N13 
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Para 

# 

Para  Title 

Requirement  and  Verification 
Criteria 

Need  # 
Relationship 

4.2.4 

Model  Format  -  Test 
Venues  (T=0) 

Test  venues 

N13 

4.2.5 

Model  Format  -  CTP 
Areas  (T=0) 

CTP  areas 

N13 

4.3 

Time  to  Complete 
(T=0) 

The  TE  RSCSP  model  shall 
provide  output  in  less  than  or 
equal  to  10  minutes. 

N13 

D.  TEST  SCHEDULING  MODEL  DEVELOPMENT 

As  illustrated  in  Figure  5,  the  next  step  in  the  systems  engineering  “V”  is 
model  development.  The  initial  plan  for  the  model  development  is  to  use  MS 
Excel  as  the  tool  of  choice.  The  TE  personnel  that  will  be  using  this  tool  currently 
use  MS  Excel  for  test  schedule  development.  Although  multiple  iterations  of  an 
MS  Excel  model  were  attempted,  MS  Excel  was  found  not  to  be  powerful  enough 
to  address  the  problem.  Therefore,  tools  meant  specifically  for  advanced 
optimization,  general  algebraic  modeling  system  (GAMS)  with  a  CPLEX  solver, 
were  used  to  address  the  problem.  The  TE  RSCSP  optimization  model  was 
generated  by  Shane  Edwards  (Edwards  2015). 
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IV.  MODEL  VERIFICATION 


As  illustrated  in  Figure  5,  the  next  step  in  the  systems  engineering  “V”  is 
model  verification.  Model  verification  is  performed  through  review  of  model  input 
files  and  through  review  of  the  manually  generated  schedules  developed  from  a 
model  output  file.  The  input  files  and  the  output  file  used  for  this  assessment  are 
from  a  model  run  provided  by  Shane  A.  Edwards  (Edwards  2015).  The  model  run 
used  for  verification  is  associated  with  a  file  provided  by  the  TE  planners. 

The  assessments  that  follow  present  verification  assessments  against  this 
model  run  and  use  the  terminology  of  MET,  NOT  MET,  PARTIALLY  MET,  and 
NOT  VERIFIED.  “MET”  means  that  the  requirement  has  been  assessed  to  have 
been  met  based  on  the  verification  criteria  and  the  data  provided.  “NOT  MET” 
means  that  the  requirement  has  been  assessed  not  to  have  been  met  based  on 
the  verification  criteria  and  the  data  provided.  “NOT  VERIFIED”  means  that  the 
requirementcannot  be  verified  based  on  the  verification  criteria  and  the  data 
provided  due  to  lack  of  data  or  inaccurate  data.  “PARTIALLY  MET”  means  that 
the  requirement  has  been  assessed  to  have  been  partially  met  based  on  the 
verification  criteria  and  the  data  provided;  an  assessment  of  PARTIALLY  MET 
indicates  that  a  portion  of  the  requirement  is  either  NOT  MET,  or  NOT 
VERIFIED. 

A.  MODEL  INPUTS  VERIFICATION 

The  model  is  verified  against  the  1  .x  model  input  requirements,  contained 
in  Table  3.  Since  there  is  a  single  set  of  input  files,  the  1.x  model  input 
requirements  apply  to  all  of  the  schedules  developed  by  this  model  run. 

In  order  to  assess  the  model  inputs  from  Appendix  A  against  the  1.x 
model  input  requirements  from  Table  3,  an  understanding  is  needed  of  the 
relationship  between  the  model  variables,  the  input  file  names,  and  the  model 
requirements.  Table  7  provides  this  relationship.  Table  7  columns  are  explained 
as: 
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•  “Input  #”  provides  numbering  for  the  Table  7  row. 

•  “Para  #  and  Title”  provides  the  relationship  to  Table  3. 

•  “Formulation  Variable”  provides  the  related  problem  formulation 
variables. 

•  “File  Name”  identifies  the  related  input  filename.  Model  input  filenames 
consist  of  MS  Excel  .csv  files  and  are  contained  in  Appendix  A. 

Table  7.  Model  Inputs  Relationship  to  Model  and  Model  Files 


Input  # 

Para  #  and  Title 

Formulation 

Variable 

File  Name 

11 

1 .1  Test  Period 

P 

p.csv 

12 

1 .2  Test  Events 

t 

t.csv 

13 

1.3  Test  Asset  Type 

a 

a.csv 

14 

1 .4  Test  Venues 

V 

v.csv 

15 

1 ,7  Test  Event  Priorities 

i 

i.csv 

16 

1 .6  Test  Venue  Movement  Test 
Periods 

m_periodsv,V’ 

m_periods.csv 

17 

1 .7  Test  Venue  Test  Asset  Capacity 

v  capv 

v  cap. csv 

18 

1 .8  Test  Event  Priority 

Relationships 

it 

ti.csv 

19 

1 .9  Test  Event  Test  Venue 
Relationships 

vt 

vt.csv 

no 

1 .1 0  Test  Event  Precedence 
Relationships 

Rt 

rt.csv 

111 

1.11  Test  Event  Test  Period 
Durations 

t_periodst 

t_data.csv 

112 

1.12  Number  of  Test  Assets 
Needed  for  Test  Events 

a_rect 

t_data.csv 

113 

1.13  Test  Asset  Type  Test  Event 
Relationships 

3_type_reqta 

ta_data.csv 

114 

1.14  Test  Asset  Test  Venue 
Starting  Location 

a_recaVgp 

a_data.csv 

115 

1 .1 5  Test  Asset  Unavailability 

unavailaxp 

a  data.csv 

116 

1 .1 6  Test  Venue  Unavailability 

unavaila.v.p 

a  data.csv 

117 

1 .1 7  Test  Event  Priority  Deadlines 

deadline , 

Ldata.csv 

118 

1.18  Priority  Deadline  Penalties 

penalty / 

Ldata.csv 
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Based  on  the  relationships  identified  in  Table  7,  the  input  files  contained  in 
Appendix  A  are  assessed  against  the  model  1.x  input  requirements  contained  in 
Table  3.  Because  there  is  a  single  set  of  input  files,  the  1.x  model  input 
requirements  apply  to  all  of  the  schedules  developed  by  this  model  run. 
Verification  assessment  results,  against  the  1.x  requirements,  are  given  in  the 
last  column,  “Verification  &  Rationale,”  of  Table  8. 


Table  8.  Model  Input  Verification 


Para 

# 

Para  Title  and  Verification  Criteria 

Verification  &  Rationale 

1.0 

Model  Inputs.  The  TE  RSCSP  model  shall 
accept  the  following  input,  which  is  verified 
by  reviewing  the  model  input  file: 

N/A 

1.1 

Test  Period  (T=0) 

MET 

p.csv  has  data 

1.2 

Test  Events  (T=0) 

MET 

t.csv  has  data 

1.3 

Test  Asset  Type  (T=0) 

MET 

a.csv  has  data* 

1.4 

Test  Venues  (T=0) 

MET 

v.csv  has  data 

1.5 

Test  Event  Priorities  (T=0) 

MET 

i.csv  has  data 

1.6 

Test  Venue  Movement  Test  Periods  (T=0) 

MET 

m_periods.csv  has 
data 

1.7 

Test  Venue  Test  Asset  Capacity  (T=0) 

MET 

v  cap. csv  has  data 

1.8 

Test  Event  Priority  Relationships  (T=0) 

MET 

ti.csv  has  data 

1.9 

Test  Event  Test  Venue  Relationships 
(T=0) 

MET 

vt.csv  has  data 

1.10 

Test  Event  Precedence  Relationships 
(T=0) 

MET 

rt.csv  has  data 

1.11 

Test  Event  Test  Period  Durations  (T=0) 

MET 

t  data.csv  has  data 

1.12 

Number  of  Test  Assets  Needed  for  Test 
Events  (T=0) 

MET 

t_data.csv  has  data 

1.13 

Test  Asset  Type  Test  Event  Relationships 
(T=0) 

MET 

ta_data.csv  has 
data 

1.14 

Test  Asset  Test  Venue  Starting  Location 
(T=0) 

MET 

a_data.csv  has  data 

1.15 

Test  Asset  Unavailability  (0) 

MET 

a  data.csv  has  data 

1.16 

Test  Venue  Unavailability  (0) 

MET 

a  data.csv  has  data 

1.17 

Test  Event  Priority  Deadlines  (T=0) 

MET 

i  data.csv  has  data 

1.18 

Priority  Deadline  Penalties  (T=0) 

MET 

a data.csv  has  data 

*Note:  Only  one  asset  type  given  in  this  example. 
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The  model  verification  assessment  for  the  1.x  model  input  requirements, 
contained  in  Table  3,  indicate  that  all  18  model  input  requirements  in  Table  8  are 
assessed  as  MET. 

B.  MODEL  MECHANISMS,  RESOURCES,  AND  TOOLS  VERIFICATION 

Next,  verification  of  the  model  against  the  3.x  model  requirements 
(mechanisms,  resources,  and  tools),  contained  in  Table  5,  is  performed.  Since 
there  is  a  single  set  of  input  and  output  files,  these  requirements  apply  to  all  of 
the  schedules  developed  by  this  model  run.  The  model  3.x  requirements 
(mechanisms,  resources,  and  tools)  are  assessed  against  the  input  and  output 
files  contained  in  Appendix  A.  Verification  assessment  results,  against  the  3.x 
requirements,  are  given  in  the  last  column,  “Verification  &  Rationale,”  of  Table  9. 


Table  9.  Model  Mechanisms,  Resources  and  Tools  Requirements 

Verification 


Para 

# 

Para  Title 

Requirement  Verification 
Criteria 

Verification  &  Rationale 

3.0 

Mechanisms, 
Resources 
and  Tools 

The  TE  RSCSP  model 
tool  requirements  shall 
be  verified  by  reviewing 
the  model  files: 

Not  Applicable 

3.1 

MS  Excel 

Input  (T=0) 

The  format  of  the  TE 
RSCSP  model  input  files 
shall  be  MS  Excel. 

MET 

3.2 

MS  Excel 

Model 

(Objective) 

TheTE  RSCSP  model 
shall  use  MS  Excel  for 
calculation,  output  and 
display  similar  to  current 
TE  output. 

NOT  MET 

GAMS  and  CMPLX  Solver  is 
used  for  implementation. 
Model  input  files  use  MS 

Excel  .csv  files. 

Results  of  the  model  verification  assessment  against  the  3.x  model 
requirements  (mechanisms,  resources,  and  tools)  indicate  that  the  one  T=0 
requirement  for  MS  Excel  input  files  is  assessed  as  MET.  The  one  3.x  objective 
requirement  for  MS  Excel  model  implementation  is  assessed  as  NOT  MET. 
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C.  MODEL  OUTPUT  VERIFICATION 

Next,  verification  of  the  model  against  the  4.x  model  output  requirements, 
contained  in  Table  6,  is  performed.  There  is  a  single  output  file  that  contains  all  of 
the  schedules  developed  by  this  model  run.  The  model  4.x  output  requirements 
are  assessed  against  the  single  output  file  reference  in  Appendix  A.  Verification 
assessment  results  against  the  4.x  requirements  are  given  in  the  last  column, 
“Verification  &  Rationale,”  of  Table  10. 


Table  10.  Model  Output  Requirement  Verification 


Para 

# 

Para  Title 

Requirement  and 

Verification  Criteria 

Verification  & 

Rationale 

4.0 

Model  Outputs 

TheTE  RSCSP  model 
output  requirements  shall 
be  verified  by  reviewing  the 
model  output  files: 

Not  Applicable 

4.1 

MS  Excel  Model 
Output  (T=0) 

The  TE  RSCSP  shall  use 

MS  Excel  for  the  model 
output. 

NOT  MET 

4.2 

Model  Format  (T=0) 

The  TE  RSCSP  Model  shall 
be  in  the  same  format  as 
the  TE  manual  method: 

The  current  output  file 
must  be  converted 
from  machine  format 
to  a  MS  Excel  human 
readable  format. 

4.2.1 

Model  Format  - 
Months  (T=0) 

Months 

NOT  MET,  See  4.2 

4.2.2 

Model  Format  -  Test 
Periods  (T=0) 

Test  periods 

NOT  MET,  See  4.2 

4.2.3 

Model  Form  -  Test 
Assets  (T=0) 

Test  assets 

NOT  MET,  See  4.2 

4.2.4 

Model  Format  -  Test 
Venues  (T=0) 

Test  venues 

NOT  MET,  See  4.2 

4.2.5 

Model  Format  -  CTP 
Areas  (T=0) 

CTP  areas 

NOT  MET,  See  4.2 

4.3 

Time  to  Complete 
(T=0) 

The  TE  RSCSP  model  shall 
provide  output  in  less  than 
or  equal  to  10  minutes. 

NOT  VERIFIED 
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Results  of  the  model  verification  assessment  against  the  4.x  model  output 
requirements,  contained  in  Table  6,  indicate  that  six  of  the  seven  T=0  model 
output  requirements  in  Table  10  are  NOT  MET.  One  of  the  seven  T=0  model 
output  requirements  (Time  to  Complete)  is  NOT  VERIFIED. 

D.  MODEL  CONTROL  AND  CONSTRAINT  REQUIREMENT 

VERIFICATION 

Lastly,  verification  of  the  model  against  the  2.x  model  requirements 
(control  and  constraint),  contained  in  Table  4,  is  performed.  These  requirements 
apply  to  each  of  the  five  schedules  developed  by  this  model  run. 

Before  the  schedules  can  be  assessed,  they  need  to  be  developed  from 
the  model  output  file.  The  model  output  file  is  given  in  the  format  of  test  venue, 
test  period,  test  event,  number  of  test  assets  assigned  to  test,  and  total  assets  in 
test,  as  shown  in  Figure  10.  This  information  is  provided  for  each  schedule. 

schedule  of  tests  completed 
ATC_MD 
pOOl 

t25  Initial  Inspection  and  Safety  started: 
pOOl  testing  4  test  assets 

4  total 

in  test 


p034 

t04  Land  Mode  Braking  started:  p034 

testing  1  test  assets 

til  Rolling  Resistance  started:  p032 

testing  1  test  assets 

tl2  Fuel  Consumptionjand  started:  p031 
testing  1  test  assets 

t40  Physical  Characteristics  started:  p031 
testing  1  test  assets 

4  total 

in  test 


Figure  10.  Example  of  Model  Output.  Source:  Edwards  (2015). 
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The  model  output  information,  for  each  of  the  five  schedules,  is  manually 
translated  from  the  model  output  into  MS  Excel  schedules,  similar  to  how  the  TE 
planner  schedules  are  generated.  The  schedule  generated  is  called  a  3-D 
schedule  because  it  contains  the  items  given  in  the  3-D  diagram  of  Figure  8:  test 
asset  (y-axis),  test  period  (x-axis),  and  test  venue  (color).  Each  of  these  detailed 
schedules  is  individually  provided  in  Appendix  B  for  each  of  the  five  schedules  in 
the  form  of  monthly  schedules  with  daily  test  information.  An  example  of  model 
output  translated  into  a  3-D  schedule  is  provided  in  Figure  1 1 . 


Test  Asset 

P041 

P042 

P043 

P044 

P045 

P046 

LM 

LM 

LM 

F 

F 

F 

TA1 

t12 

t12 

t12 

t64 

t64 

t64 

LM 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

TA2 

t59 

t59 

t59 

t59 

t59 

t59 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

TA3 

t40 

t40 

t40 

X40 

t40 

t40 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

TA4 

t28 

t28 

t28 

t28 

t28 

t28 

S 

S 

S 

S 

S 

S 

TA5 

t53 

t53 

t53 

t53 

t53 

t53 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

TA6 

t58 

t58 

t58 

t58 

t58 

t58 

WM 

WM 

WM 

S/HF 

S/HF 

S/HF 

TA7 

t15 

t15 

t15 

t37 

t37 

t37 

Figure  1 1 .  Example  of  Generated  3-D  Detailed  Monthly  Schedule 


In  the  manually  generated  3-D  detailed  monthly  schedules: 

•  Test  assets  are  given  on  the  y-axis  as  test  asset  1  (TA1)  through  test 
asset  7  (TA7). 
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•  Test  periods  are  given  on  the  x-axis  as  test  period  1  (P001)  through 
test  period  xyz  (Pxyz). 

•  The  cells  where  the  x  and  y  axis  intersect  are  colored  based  on  test 
venue:  Amphibious  Vehicle  Test  Branch  (AVTB)  =  blue,  Aberdeen  Test 
Center  (ATC)  =  green,  Yuma  Proving  Ground  (YPG)  =  purple,  and 
White  Sands  Missile  Range  (WSMR)  =  salmon. 

•  The  interior  cells  contain:  the  associated  CTP  area,  the  test  event,  and 
the  test  priority  indicated  by  text  font. 

•  The  top  line  of  the  cells  in  the  interior  of  the  table  displays  the 
functional  test  CTP  area:  LM  =  land  mobility,  F  =  firepower,  RGT1  = 
reliability  growth  testing  1,  S/HF  =  safety/human  factors,  WM  =  water 
mobility,  and  S  =  survivability. 

•  The  entire  CTP  area  list  is  given  in  Table  14.  The  CTP  area  was  not 
used  by  the  model,  but  is  provided  in  this  manually  generated  set  of 
schedules. 

•  The  second  line  of  the  ceils  in  the  interior  of  the  table  contains  the 

actual  test  event  that  is  performed  on  that  test  asset  during  that  time 
period.  Examples  of  test  events  are  t12  =  fuel  consumption,  t64  = 
firing,  t59  =  RGT,  t40  =  physical  characteristics,  t28  =  land  mode  ride 
quality,  t53  =  electromagnetic  environmental  effects  (E3) 

testingjimited,  t58  =  RGT,  t15  =  plow  in  testing,  and  t37  =  automotive 
toxic  fumes  in  water.  The  entire  list  of  test  events  for  this  model  run  is 
located  in  Appendix  A,  Table  31 . 

•  The  cells  in  the  interior  of  the  table  are  given  a  font  based  on  priority: 
high  =  bold,  medium  =  italic,  and  low  =  normal. 

As  an  example,  test  asset  TA1  in  test  period  P041  is  performing  a  high- 
priority,  fuel-consumption  test  event,  with  a  CTP  area  of  land-mobility  at  test 
venue  ATC.  TA1  continues  this  testing  in  P042  and  P043.  TA1  transitions  into  a 
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high-priority,  firing  test  event,  with  a  CTP  area  of  firepower  in  P044.  TA1 
continues  this  testing  into  P045  and  P046.  As  another  example,  test  asset  TA3 
performs  a  medium-priority,  physical  characteristics  test  event,  with  a  CTP  area 
of  safety  and  human  factors  at  test  venue  ATC  for  test  periods  T041  through 
T046. 

The  model  output  manually  generated  3-D  detailed  monthly  schedules, 
contained  in  Appendix  B,  are  assessed  against  the  model  requirements  (control 
and  constraint).  Based  on  these  five  schedule  assessments,  results  are  given  in 
the  last  column,  “Verification  &  Rationale,”  of  Table  1 1 . 


Table  1 1 .  Model  Control  and  Constraint  Requirements  Verification 


Para 

# 

Requirement  and  Verification  Criteria 

Verification  & 
Rationale 

2.0 

Controls  and  Constraints  (T=0).  The  TE 
RSCSP  model  shall  accept  and  use 
model  controls  and  constraints,  which  are 
verified  by  reviewing  the  model  files  and 
3-D  conversions: 

The  verification 
assessment  is 
the  same  for  all 
five  schedules: 

2.1 

Test  Period  (T=0).  Schedule  is  within  the 
test  period  given  in  the  test  period  input 
file. 

MET 

2.1.1 

One  Test  Event  on  Test  Asset  during 

Time  Period  (T=0).  Schedule  does  not 
show  more  than  one  test  event  in  a  time 
period  on  a  test  asset. 

PARTIALLY  MET 

2.1.2 

Place  all  Test  Events  in  Time  Period 
(T=0).  Schedule  places  all  identified  test 
events  in  a  test  period. 

MET 

2.1.3 

Test  Event  Test  Period  Durations  (T=0). 
Schedule  test  event  test  period  durations 
matches  the  test  events  test  period  input 
file. 

MET 

2.2 

Test  Events  (T=0).  Schedule  includes  all 
test  events  included  in  the  test  events 
input  file. 

MET 

2.3 

Test  Asset  Type  (0).  Schedule  results 
places  test  events  on  the  applicable  test 
asset  type  based  on  the  input  file. 

NOT  VERIFIED 
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Para 

# 

Requirement  and  Verification  Criteria 

Verification  & 
Rationale 

2.4 

Test  Venues  (T=0).  Schedule  includes 
only  the  test  venues  included  in  the  test 
venues  input  file. 

MET 

2.5 

Test  Event  Priority 

NOT 

APPLICABLE 

2.5.1 

Test  Event  Priority  Relationships  (T=0). 
Schedule  results  are  constrained  by  the 
test  event  priority  relationships  input  file 

PARTIALLY  MET 

2.5.2 

High  to  Medium  Priority  Relationship 
(T=0).  Schedule  starts  high  priority  test 
events  before  medium  priority  test  events. 

PARTIALLY  MET 

2.5.3 

Medium  to  Low  Priority  Relationship 
(T=0).  Schedule  starts  medium  priority 
test  events  before  low  priority  test  events. 

NOT  VERIFIED 

2.5.4 

High  to  Low  Priority  Relationships  (T=0). 
Schedule  starts  high  priority  test  events 
before  low  priority  test  events. 

NOT  VERIFIED 

2.6 

Test  Venue  Movement  Test  Periods 
(T=0).  Schedule  uses  the  test  venue 
movement  test  periods  based  on  the  input 
file. 

MET 

2.6.1 

Add  Time  Period  from  Test  Venue 
Movement  (T=0).  Schedule  shows  that 
the  time  periods  added  to  the  schedule 
when  the  test  asset  moves  to  a  new  test 
venue  are  based  on  the  input  file. 

MET 

2.6.2 

Schedule  Test  Event  on  Available  Test 
Venues  (T=0).  Schedule  shows  that  the 
test  events  are  scheduled  only  on 
available  test  venues  based  on  the  input 
file. 

MET 

2.7 

Test  Venue  Test  Asset  Capacity  (T=0). 

The  number  of  test  assets  located  at  each 
test  venue  on  the  schedule  does  not 
exceed  the  capacity  identified  in  the  input 
file. 

NOT  VERIFIED 

2.8 

Test  Asset  Test  Venue  Starting  Location 
(T=0).  Schedule  test  assets  start  at  the 
venues  identified  in  the  test  asset  test 
venue  input  file. 

MET 
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Para 

# 

Requirement  and  Verification  Criteria 

Verification  & 
Rationale 

2.9 

Test  Event  Test  Venue  Relationships 
(T=0).  Schedule  test  events  are 
performed  at  the  test  venues  identified  in 
the  test  event  test  venue  relationships 
input  file. 

MET 

2.10 

Test  Event  Precedence  Relationships 
(T=0).  Schedule  is  constrained  by  the  test 
event  precedence  relationships  input  file. 

MET 

2.10.1 

Test  Event  Predecessor  First  (T=0). 
Predecessor  test  events  are  on  the 
schedule  before  successor  test  events 
regardless  of  test  event  priority. 

MET 

2.10.2 

Test  Event  Successor  Second  (T=0). 
Successor  test  events  are  on  the 
schedule  before  predecessor  test  events 
regardless  of  test  event  priority. 

MET 

2.11 

Number  of  Test  Assets  Needed  for  Test 
Events  (T=0).  Schedule  uses  the  number 
of  test  assets  needed  for  test  events  input 
file. 

MET 

2.12 

Low  Priority  Schedule  Relationship  (0). 
Schedule  results  match  the  input  file. 

NOT  VERIFIED 

2.13 

Test  Asset  Unavailability  (0).  Schedule 
places  test  assets  only  on  available  test 
assets  based  on  the  input  file. 

NOT  VERIFIED 

2.14 

Test  Venue  Unavailability  (0).  Schedule 
results  are  constrained  by  the  input  file. 

NOT  VERIFIED 

2.15 

Test  Event  Priority  Deadlines  (T=0). 
Schedule  results  are  constrained  by  the 
input  file. 

PARTIALLY  MET 

2.15.1 

High  Priority  to  Deadline  Relationship 
(T=0)  Schedule  results  start  high  priority 
test  events  before  the  high  priority 
deadline. 

MET 

2.15.2 

Medium  Priority  to  Deadline  Relationship 
(T=0).  Schedule  results  start  medium 
priority  test  events  before  the  medium 
priority  deadline. 

NOT  VERIFIED 

2.15.3 

High  Priority  to  Test  Period  Relationship 
(T=0).  Schedule  results  complete  high 
priority  test  events  before  the  test  period 
completes. 

MET 
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Para 

# 

Requirement  and  Verification  Criteria 

Verification  & 
Rationale 

2.15.4 

Medium  Priority  to  Test  Period 
Relationship  (T=0).  Schedule  results 
complete  medium  priority  test  events 
before  the  test  period  completes. 

MET 

Results  of  the  model  verification  assessment  against  the  2.x  model  output 
requirements  (control  and  constraint),  contained  in  Table  4,  indicate  that  17  of 
the  25  T=0  model  output  requirements  in  Table  11  are  MET,  four  are 
PARTIALLY  MET,  and  four  are  NOT  VERIFIED.  Four  of  the  four  Objective  2.x 
model  requirements  (control  and  constraint)  are  NOT  VERIFIED. 

E.  MODEL  VERIFICATION  SYNOPSIS 

The  detailed  verification  performed  against  the  model  is  synopsized, 
showing  verification  against  all  of  the  model  schedules,  and  is  provided  in  Table 
12.  The  model  2.x  requirements  (control  and  constraint)  against  each  of  the 
model  schedules  (beta_max,  t_periods,  betajmin,  betajmode  and  beta_mean) 
have  resulted  in  the  same  assessment  against  all  of  them;  therefore,  only  one 
verification  result  is  provided  in  the  Table  12. 


Table  12.  Synopsized  Model  Verification  Results 


Para 

# 

Para  Title 

Verification 

1.0 

Model  Inputs 

N/A 

1.1 

Test  Period  (T=0) 

MET 

1.2 

Test  Events  (T=0) 

MET 

1.3 

Test  Asset  Type  (T=0) 

MET 

1.4 

Test  Venues  (T=0) 

MET 

1.5 

Test  Event  Priorities  (T=0) 

MET 

1.6 

Test  Venue  Movement  Test  Periods  (T=0) 

MET 

1.7 

Test  Venue  Test  Asset  Capacity  (T=0) 

MET 

1.8 

Test  Event  Priority  Relationships  (T=0) 

MET 

1.9 

Test  Event  Test  Venue  Relationships  (T=0) 

MET 

1.10 

Test  Event  Precedence  Relationships  (T=0) 

MET 
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Para 

# 

Para  Title 

Verification 

1.11 

Test  Event  Test  Period  Durations(T=0) 

MET 

1.12 

Number  of  Test  Assets  Needed  for  Test  Events 
(T=0) 

MET 

1.13 

Test  Asset  Type  Test  Event  Relationships  (T=0) 

MET 

1.14 

Test  Asset  Test  Venue  Starting  Location  (T=0) 

MET 

1.15 

Test  Asset  Unavailability  (0) 

MET 

1.16 

Test  Venue  Unavailability  (O) 

MET 

1.17 

Test  Event  Priority  Deadlines  (T=0) 

MET 

1.18 

Priority  Deadline  Penalties  (T=0) 

MET 

2.0 

Controls  and  Constraints 

N/A 

2.1 

Test  Period  (T=0) 

MET 

2.1.1 

One  Test  Event  on  Test  Asset  durinq  Time  Period 
(T=0) 

PARTIALLY  MET 

2.1.2 

Place  all  Test  Events  in  Test  Period  (T=0) 

MET 

2.1.3 

Test  Event  Test  Period  Durations  (T=0) 

MET 

2.2 

Test  Events  (T=0) 

MET 

2.3 

Test  Asset  Type  (0) 

NOT  VERIFIED 

2.4 

Test  Venues  (T=0) 

MET 

2.5 

Test  Event  Priorities 

N/A 

2.5.1 

Test  Event  Priority  Relationships  (T=0) 

PARTIALLY  MET 

2.5.2 

High  to  Medium  Priority  Relationship  (T=0) 

PARTIALLY  MET 

2.5.3 

Medium  to  Low  Priority  Relationship  (T=0) 

NOT  VERIFIED 

2.5.4 

High  to  Low  Priority  Relationships  (T=0) 

NOT  VERIFIED 

2.6 

Test  Venue  Movement  Test  Periods  (T=0) 

MET 

2.6.1 

Add  Time  Period  from  Test  Venue  Movement 
(T=0) 

MET 

2.6.2 

Schedule  Test  Event  on  Available  Test  Venues 
(T=0) 

MET 

2.7 

Test  Venue  Test  Asset  Capacity  (T=0) 

NOT  VERIFIED 

2.8 

Test  Asset  Test  Venue  Starting  Location  (T=0) 

MET 

2.9 

Test  Event  Test  Venue  Relationships  (T=0) 

MET 

2.10 

Test  Event  Precedence  Relationships  (T=0) 

MET 

2.10.1 

Test  Event  Predecessor  First  (T=0) 

MET 

2.10.2 

Test  Event  Successor  Second  (T=0) 

MET 

2.11 

Number  of  Test  Assets  Needed  for  Test  Events 
(T=0) 

MET 

2.12 

Low  Priority  Schedule  Relationship  (0) 

NOT  VERIFIED 

2.13 

Test  Asset  Unavailability  (0) 

NOT  VERIFIED 

2.14 

Test  Venue  Unavailability  (0) 

NOT  VERIFIED 

2.15 

Test  Event  Priority  Deadlines  (T=0) 

PARTIALLY  MET 

2.15.1 

High  Priority  to  Deadline  Relationship  (T=0) 

MET 

2.15.2 

Medium  Priority  to  Deadline  Relationship  (T=0) 

NOT  VERIFIED 

2.15.3 

High  Priority  to  Test  Period  Relationship  (T=0) 

MET 
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Para 

# 

Para  Title 

Verification 

2.15.4 

Medium  Priority  to  Test  Period  Relationship  (T=0) 

MET 

3.0 

Mechanisms,  Resources  and  Tools 

N/A 

3.1 

MS  Excel  Input  (T=0) 

MET 

3.2 

MS  Excel  Model  (T=0) 

NOT  MET 

4.0 

Model  Outputs 

N/A 

4.1 

MS  Excel  Model  Output  (T=0) 

NOT  MET 

4.2 

Model  Format  (T=0) 

NOT  MET 

4.2.1 

Model  Format  -  Months  (T=0) 

NOT  MET 

4.2.2 

Model  Format  -  Test  Periods  (T=0) 

NOT  MET 

4.2.3 

Model  Form  -  Test  Assets  (T=0) 

NOT  MET 

4.2.4 

Model  Format  -  Test  Venues  (T=0) 

NOT  MET 

4.2.5 

Model  Format  -  CTP  Areas  (T=0) 

NOT  MET 

4.3 

Time  to  Complete  (T=0) 

NOT  VERIFIED 
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V.  MODEL  VALIDATION 


As  illustrated  in  Figure  5,  the  next  step  in  the  systems  engineering  “V”  is 
validation.  For  model  validation,  the  model  is  assessed  from  multiple 
perspectives.  The  model  is  assessed  in  a  combined  DT/OT  event  using  the 
model  developer’s  verification  model  inputs  and  outputs  to  assess  the  model 
from  a  validation  perspective  against  the  fourteen  operational  needs  identified  in 
Table  1.  This  assessment  is  followed  by  an  operational  assessment  (OA)  of  the 
model  schedules  against  PM  AAA  TE  developed  schedules  for  operational 
effectiveness  (OE)  and  operational  suitability  (OS).  This  validation  information  is 
then  aggregated  against  the  COIs  for  assessment  at  a  higher  level. 

A.  VALIDATION  AGAINST  OPERATIONAL  NEEDS 

To  perform  validation  assessment  against  the  required  capabilities,  also 
known  as  operational  needs,  we  look  at  the  verification  results  relative  to  the 
operational  needs,  taking  a  step  back  from  the  deep  dive  of  detailed 
requirements.  The  assessment  of  the  operational  needs  relative  to  model 
verification  results  is  provided  in  Table  13. 
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Table  13.  Operational  Needs  and  Model  Verification  Results 


Need  Title  Tier  Operational  Model  Verification  Assessment 


# 

N1 


Asset  Types 


(O) 


Needs 


APA  The  TE  NOT  VERIFIED.  Model  data  did  not  include  data  that  would  verify  this. 

RSCSP  model  -  NOT  VERIFIED  -  2.1 2  -  Test  Asset  Type  Test  Event  Relationships  (O) 
low  priority  test 
events  shall  be 
allowed  to  go 
beyond  the  test 


N2 


N3 


period. 


Multiple 

Assets 

(T=0) 

KSA  TheTE 

RSCSP  model 
shall  allow 
multiple  test 
assets  to  be 
tested 

simultaneously. 

PARTIALLY  MET.  Although  the  model  includes  the  constraint,  model  data 
did  not  include  data  that  would  verify  that  multiple  test  asset  variants  are 
included.  For  a  single  test  asset  type,  this  requirement  is  MET. 

-  MET  1 .3  -  Test  Asset  Type  (0) 

-  MET  1.12-  Number  of  Test  Assets  Needed  for  Test  Events  (T=0) 

-  MET  1.13  -  Test  Asset  Type  Test  Event  Relationships  (T=0) 

-  NOT  VERIFIED  -  2.3  -  Test  Asset  Type  (0) 

Time  Period 

APA  TheTE 

NOT  VERIFIED.  Although  the  model  includes  the  constraint,  model  data 

Asset 

RSCSP  model 

did  not  include  data  that  would  verify  this. 

Availability 

shall  have  a 

-  MET  - 1 .1 5  -  Test  Asset  Unavailability  (0) 

(0) 

time  period  test 

-  NOT  VERIFIED  -  2.1 3  -  Test  Asset  Unavailability  (0) 

asset 

availability 

constraint. 
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Need 

# 

Title 

Tier 

Operational 

Needs 

Model  Verification  Assessment 

N4 

Venue 

Distance 

(T=0) 

KSA 

The  TE 

RSCSP  model 
shall  minimize 
test  movement 
between  test 

venues. 

MET.  Test  venue  distance  is  addressed. 

-  MET  -  2.6  -  Test  Venue  Movement  Test  Periods  (T=0) 

-  MET  -  2.6.1  -  Add  Time  Period  from  Test  Venue  Movement  (T=0) 

N5 

Event 

Priority 

Placement 

(T=0) 

KSA 

The  TE 

RSCSP  model 
shall  place  test 
events  based 
on  priority. 

PARTIALLY  MET.  Some  test  events  of  lower  priority  are  tested  before 
higher  priorities. 

-  MET  - 1 .2  -  Test  Events  (T=0) 

-  MET  - 1.5  -  Test  Event  Priorities  (T=0) 

-  MET  - 1 .8  -  Test  Event  Priority  Relationships  (T=0) 

-  PARTIALLY  MET  -  2.1 .1  -  One  Test  Event  on  Test  Asset  during  Time 
Period  (T=0) 

-  MET  -  2.1.2  -  Place  all  Test  Events  in  Test  Period  (T=0) 

-  MET  -  2.2  -  Test  Events  (T=0) 

-  PARTIALLY  MET  -  2.5.1  -  Test  Event  Priority  Relationships  (T=0) 

-  PARTIALLY  MET  -  2.5.2  -  High  to  Medium  Priority  Relationship  (T=0) 

-  NOT  VERIFIED  -  2.5.3  -  Medium  to  Low  Priority  Relationship  (T=0) 

-  NOT  VERIFIED  -  2.5.4  -  High  to  Low  Priority  Relationships  (T=0) 

-  NOT  VERIFIED  -  2.15.2  -  Medium  Priority  to  Deadline  Relationship  (T=0) 

-  MET  -  2.15.3  -  High  Priority  to  Test  Period  Relationship  (T=0) 

-  MET  -  2.15.4  -  Medium  Priority  to  Test  Period  Relationship  (T=0) 
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Tier  Operational 
Needs 


APA  TheTE 

RSCSP  model 
shall  allow 
multiple  test 
venues  to  be 
used  at  the 
same  time. 

APA  TheTE 

RSCSP  model 
shall  allow  test 
venue  choice 
by  test  event. 


APA  TheTE 

RSCSP  model 
shall  allow 
multiple  test 
events  to  occur 
at  the  same 
time, 


Model  Verification  Assessment 


MET. 

-  MET  - 1 .7  -  Test  Venue  Test  Asset  Capacity  (T=0) 

-  MET  - 1 .1 6  -  Test  Venue  Unavailability  (O) 


MET.  Some  of  the  specification  requirements  that  connect  to  this  are 
derived. 

-  MET  - 1 .4  -  Test  Venues  (T=0) 

-  MET  - 1 .9  -  Test  Event  Test  Venue  Relationships  (T=0) 

-  MET  - 1 .14  -  Test  Asset  Test  Venue  Starting  Location  (T=0) 

-  MET  -  2.4  -  Test  Venues  (T=0) 

-  MET  -  2.6.2  -  Schedule  Test  Event  on  Available  Test  Venues  (T=0) 

-  NOT  VERIFIED  -  2.7  -  Test  Venue  Test  Asset  Capacity  (T=0) 

-  MET  -  2.8  -  Test  Asset  Test  Venue  Starting  Location  (T=0) 

-  MET  2.9  -  Test  Event  Test  Venue  Relationships  (T=0) 

-  NOT  VERIFIED  -  2.14  -  Test  Venue  Unavailability  (O) 

MET 

-  MET  -  2.1 1  -  Number  of  Test  Assets  Needed  for  Test  Events  (T=0) 
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Need 

# 

N9 


N10 


Nil 


N12 


Title 

Tier 

Operational 

Needs 

Model  Verification  Assessment 

Precedence 

APA 

The  TE 

MET. 

Relationship 

RSCSP  model 

-  MET  - 1.10  -  Test  Event  Precedence  Relationships  (T=0) 

(T=0) 

shall  have  test 
event 

predecessor 
and  successor 
relationship 
constraints. 

-  MET  -  2.10  -  Test  Event  Precedence  Relationships  (T=0) 

-  MET  -  2.10.1  -  Test  Event  Predecessor  First  (T=0) 

-  MET  -  2.10,2  -  Test  Event  Successor  Second  (T=0) 

Deadline  for 

APA 

The  TE 

PARTIALLY  MET.  This  requirement  is  not  completely  verified.  Only  high 

each  Priority 

RSCSP  model 

priority  deadlines  are  verified. 

(T=0) 

shall  place  test 
assets  based 
on  a  deadline 
for  each 
priority  type. 

-  MET  - 1.17  -  Test  Event  Priority  Deadlines  (T=0) 

-  MET  - 1.18  -  Priority  Deadline  Penalties  (T=0) 

-  PARTIALLY  MET  -  2.15  -  Test  Event  Priority  Deadlines  (T=0) 

-  MET  -  2.15.1  -  High  Priority  to  Deadline  Relationship  (T=0) 

Schedule 

KPP 

The  TE 

MET. 

Test  Events 

RSCSP  model 

-  MET  -1.1  -  Test  Period  (T=0) 

(T=0) 

shall  minimize 
the  time  period 
used  for  test 
events. 

-  MET  -1.11  -  Test  Event  Test  Period  Durations  (T=0) 

-  MET  -  2.1  -  Test  Period  (T=0) 

-  NOT  VERIFIED  -  2.1 .4  -  Test  Venue  Unavailability  (T=0) 

MS  Excel 

APA 

The  TE 

PARTIALLY  MET.  While  MS  Excel  input  files  are  used,  data  is  not  pulled 

Input  (T=0) 

RSCSP  model 
shall  allow  TE 
personnel  to 
input 

information  in 
MS  Excel. 

directly  from  MS  Excel.  Some  work  and  know-how  is  required  to  create 
input  files. 

-  MET  -  3.1  -  MS  Excel  Input  (T=0) 
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Excel  (O) 


Operational 

Needs 


TheTE 
RSCSP  model 
shall  output 
schedules  in  a 
MS  Excel 
Format 
showing  each 
vehicle,  using 
colors  for  test 
site,  and  using 
blocks  that 
aggregate  the 
information 
into  the  month. 
The  TE 
RSCSP  model 
shall  use  MS 
Excel  only. 
Note:  This 
means  that  a 
specialized 
tool  will  not  be 
used  for  the 
model. 


Model  Verification  Assessment 


NOT  MET.  This  output  is  generated  manually. 

-  NOT  MET  -  4.1  -  MS  Excel  Model  Output  (T=0) 

-  NOT  MET  -  4.2.1  -  Model  Format  -  Months  (T=0) 

-  NOT  MET  -  4.2.2  -  Model  Format  -  Test  Periods  (T=0) 

-  NOT  MET  -  4.2.3  -  Model  Format  -  Test  Assets  (T=0) 

-  NOT  MET  -  4.2.4  -  Model  Format  -  Test  Venues  (T=0) 

-  NOT  MET  -  4.2.5  -  Model  Format  -  CTP  Areas  (T=0) 

-  NOT  VERIFIED  -  4.3  -  Time  to  Complete  (T=0) 


NOT  MET.  The  GAMS  programming  tool  is  used. 
-  NOT  MET  -  3.2  -  MS  Excel  Model  (O) 
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Results  follow  of  the  DT/OT  validation  assessment  of  the  capabilities: 

•  One  of  two  KPPs  is  assessed  as  MET. 

•  One  of  three  KSAs  is  assessed  as  MET. 

•  Four  of  six  Threshold  =  Objective  APAs  are  assessed  as  MET. 

•  Zero  of  three  Objective  APAs  are  assessed  as  MET. 

B.  TE  PLANNING  PROCESS  SCHEDULES 

The  DT  validation  assessment  against  the  capabilities  is  augmented  by  a 
validation  assessment  of  the  model  results  against  a  similar  TE  schedule 
development  effort.  The  assessment  uses  a  test  schedule  for  a  particular  PM 
AAA  program.  Per  Step  1  of  Figure  6,  the  TE  planner  identifies  what  test  events 
are  required  along  with  the  number  of  days  needed  for  each  test  event  by 
reviewing  the  system  performance  specification.  This  information  is  provided  in 
the  form  of  a  spreadsheet  in  Table  14  and  is  given  by  the  “Test  Plan/Sheet”  and 
“Test  Days  Required”  columns. 

Per  steps  2  and  3  of  Figure  6,  the  PMO  identifies  the  number  of  test 
assets  and  the  test  period  for  a  particular  course  of  action.  For  the  example  used 
in  this  model  run,  the  TE  planners  use  seven  vehicles  and  a  maximum  schedule 
length  of  190  days.  This  schedule  length  equates  to  about  nine  months, 
depending  on  how  many  days  are  used  in  the  calculation  of  a  month  of  test 
events.  This  assessment  uses  20  days  per  month. 

Per  Step  4  of  Figure  6,  the  TE  planner  updates  the  “Test  Days  Required” 
for  the  test  event  to  a  prior  TE  schedule  planning  spreadsheet.  The  other  four 
columns  of  the  TE  spreadsheet  are  generated  based  on  calculations  relative  to 
the  “Test  Days  Required”  column  as  shown  in  Table  14. 

The  “Test  Days  Required”  column  equates  to  100%  test  asset  availability. 
The  “Pessimistic”  column  is  based  on  a  30%  test  asset  availability  rate  (“Test 
Days  Required”/0.3).  The  “Most  Likely”  column  is  based  on  a  50%  test  asset 
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availability  rate  (“Test  Days  Required”/0.5).  The  “Optimistic”  column  is  based  on 
a  70%  test  asset  availability  rate  (“Test  Days  Required”/0.7).  The  “Planning 
Estimate”  column  is  based  on  the  Pessimistic,  Most  Likely,  and  Optimistic 
columns  (Pessimistic  +  Optimistic  +  4  x  “Test  Days  Required”/6). 

The  test  events  are  identified  for  the  entire  test  schedule  for  three  different 
options  by  adding  three  columns  of  Best,  Medium  and  Worst  and  through  the  use 
of  “X”s  in  the  three  columns,  as  shown  in  Table  14. 

Per  Step  5  of  Figure  6,  the  TE  planner  identifies  the  CTP  area  for  each 
test  event,  precedence,  test  venues,  and  priorities,  which  is  the  first  set  of 
columns  in  Table  14. 

The  detailed  test  schedule  planning  for  the  specific  schedule  used  to 
develop  this  model  run  is  provided  in  Table  14. 

All  of  the  tables  and  figures  provided  in  this  section  are  based  on  data 
provided  by  the  PM  AAA  TE  group  (Louis  R.  Ferguson  and  Albert  B.  Hanneman, 
email  message  to  author,  April  1, 2015). 
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Table  14.  TE  Detailed  Schedule  Planning  with  CTP  Area,  Precedence,  Priority  and  Test  Venue 


CTP 

Test 

Prerequis 

Test 

Priority 

Test  Days 

Pessi- 

Most 

Opti- 

Planning 

Best 

Medium 

Worst 

Area 

Plan/Sheet 

ite  test 

Venue 

of  Tests 

Required 

mistic 

Likely 

mistic 

Est 

LM  Tilt  Table 


LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 


Side  Slopes 
Standard 
Obstacles 
Land  Mode 
Braking 
Controlled 
Maneuverabilit 

y 

Acceleration, 
Max.  &  Min 
Speeds 
Land  Mobility 
Towing 
Longitudinal 
Slopes 
Drawbar  Pull 
and  Cooling 
System  Heat 
Balance 
Dead  Engine 
Braking 

Rolling 

Resistance 


Fuel 

LM  Consumption  - 
land 


Y  side 
slopes 


Y  Draw 
bar  pull 
test 


ATC 

ATC 

ATC 

ATC 

ATC 

ATC 

ATC 

ATC 

ATC 

ATC 

ATC 

ATC 


H 

H 

H 

H 

H 

H 

M 

H 

M 

H 

M 

H 


1 

2 

4 

2 

2 


3 

7 

13 

7 


2 

4 

8 

4 


1 

3 

6 

3 


2 

3 

10 


7 

10 

33 


4 

6 


3 

4 


20  14 


2 

2 


7 

7 


4 

4 


3 

3 


13  8 


2 

4 

9 

4 


4 

6 


21 


4 

4 
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CTP 

Test 

Prerequis 

Test 

Priority 

Test  Days 

Pessi- 

Most 

Opti- 

Planning 

Best 

Medium 

Worst 

Area 

Plan/Sheet 

ite  test 

Venue 

of  Tests 

Required 

mistic 

Likely 

mistic 

Est 

Initial 

S/HF 

Inspection  and 
Safety 

Yall  WM 
test 

AVTB 

H 

9 

30 

18 

13 

19 

X 

X 

X 

Checkout 

Fuel 

WM 

Consumption  - 

AVTB 

H 

5 

17 

10 

7 

11 

X 

X 

amphibious 

Y 

speed/po 

wering, 

controlled 

WM 

Plow  in  testing 

maneuver 

AVTB 

H 

2 

7 

4 

3 

4 

X 

X 

X 

ability, 

max 

Gross 

vehicle 

Controlled 

WM 

Maneuverabilit 

AVTB 

H 

2 

7 

4 

3 

4 

X 

X 

X 

WM 

y 

Speed  and 
Powering 
Maximum 

AVTB 

H 

2 

7 

4 

3 

4 

X 

X 

X 

WM 

Gross  Vehicle 

AVTB 

H 

2 

7 

4 

3 

4 

X 

X 

Weight 

WM 

Line  Passing 
&  Towing 

AVTB 

H 

1 

3 

2 

1 

2 

X 

X 

X 

WM 

Surf  Transit 
Maneuver  and 

AVTB 

H 

1 

3 

2 

1 

2 

X 

X 

X 

WM 

Control  with 
Weapon 

AVTB 

M 

4 

13 

8 

6 

9 

X 

Station 

WM 

Ship 

Operations 

AVTB 

H 

2 

7 

4 

3 

4 

X 

X 

X 

WM 

Multi-vehicle 

operations 

AVTB 

M 

3 

10 

6 

4 

6 

X 

X 

X 

64 


CTP 

Test 

Prerequis 

Test 

Priority 

Test  Days 

Pessi- 

Most 

Opti- 

Planning 

Best 

Medium 

Worst 

Area 

Plan/Sheet 

ite  test 

Venue 

of  Tests 

Required 

mistic 

Likely 

mistic 

Est 

Navigation 

WM 

and  Obstacle 

AVTB 

M 

3 

10 

6 

4 

6 

Avoidance 

Initial 

S/HF 

Inspection  and 
Safety 
Checkout 

Y  all  test 

ATC 

H 

9 

30 

18 

13 

19 

X 

X 

X 

Driver 

S/HF 

Visibility/Crew 
Station  Vision 

ATC 

H 

1 

3 

2 

1 

2 

X 

on  Land 

Land  Mode 

S/HF 

Ingress/Egres 

ATC 

H 

1 

3 

2 

1 

2 

X 

X 

X 

S/HF 

Land  Mode 
Ride  Quality 
Land  Mode 

ATC 

H 

5 

17 

10 

7 

11 

X 

X 

X 

S/HF 

Interior  Noise 
and  Whole 
Body  Vibration 

ATC 

H 

5 

17 

10 

7 

11 

X 

X 

X 

Driver 

S/HF 

Visibility/Crew 
Station  Vision 

AVTB 

H 

1 

3 

2 

1 

2 

X 

in  Water 

S/HF 

Ride  Quality  in 
Water 

AVTB 

H 

4 

13 

8 

6 

9 

Interior  Noise 

S/HF 

and  Whole 
Body  Vibration 

AVTB 

H 

11 

37 

22 

16 

23 

X 

in  Water 
Emergency 

S/HF 

Egress  (land 
and  water) 

AVTB 

H 

2 

7 

4 

3 

4 

X 

X 

X 

S/HF 

Maximum  Air 
Flow 

ATC 

H 

1 

3 

2 

1 

2 

X 
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CTP 

Test 

Prerequis 

Test 

Priority 

Test  Days 

Area 

Plan/Sheet 

ite  test 

Venue 

of  Tests 

Required 

Y  Man 

S/HF 

Firepower 
Toxic  Fumes 

Gun  & 
water 
gunnery 
testing 

ATC 

H 

10 

Land  Mode 

S/HF 

Automotive 
Toxic  Fumes 
Automotive 

ATC 

H 

5 

S/HF 

Toxic  Fumes 

AVTB 

H 

5 

in  Water 

S/HF 

APU  Noise 

ATC/ 

AVTB 

M 

2 

S/HF 

Transportabilit 

y 

ATC 

M 

20 

Physical 

Y 

S/HF 

Characteristic 

Transport 

ATC 

M 

5 

s 

ability 

S/HF 

Climatic 

Chambers 

Y  Man 

ATC/ 

YPG 

M 

20 

Fire  Control 

Gun  & 
water 
gunnery 

ATC 

H 

1 

r 

Inhibit  Zones 

testing 

Y  Man 

Fire  Control 

Gun  & 

F 

System  Check 

water 

ATC 

H 

15 

out 

gunnery 

testing 


66 


Pessi¬ 

mistic 


Medium 


Worst 


Most 

Opti- 

Planning 

Best 

Likely 

mistic 

Est 

33 

20 

14 

21 

X 

17 

10 

7 

11 

X 

X 

17 

10 

7 

11 

X 

X 

7 

4 

3 

4 

67 

40 

29 

43 

X 

X 

17 

10 

7 

11 

X 

X 

X 

67 

40 

29 

43 

X 

X 

3 

2 

1 

2 

50 


30 


21 


32 


CTP 

Test 

Prerequis 

Test 

Priority 

Test  Days 

Pessi- 

Most 

Opti- 

Planning 

Best 

Medium 

Worst 

Area 

Plan/Sheet 

ite  test 

Venue 

of  Tests 

Required 

mistic 

Likely 

mistic 

Est 

Y  Man 

F 

Tracking  / 
Lead 

Gun  & 
water 

ATC 

H 

5 

17 

10 

7 

11 

gunnery 

testing 

Y  Man 

Stabilization 

Gun  & 

F 

System 

water 

ATC 

H 

3 

10 

6 

4 

6 

Performance 

gunnery 

testing 

Sight  / 

F 

Boresight 
Retention 
Main  Gun 

ATC 

M 

3 

10 

6 

4 

6 

F 

testing 

(Gunner  firing) 

ATC 

H 

15 

50 

30 

21 

32 

Main  Gun 

F 

testing  (VC 

ATC 

H 

8 

27 

16 

11 

17 

firing) 

F 

COAX  testing 
(Gunner  firing) 

ATC 

H 

2 

7 

4 

3 

4 

F 

COAX  testing 

ATC 

H 

2 

7 

4 

3 

4 

(VC  Firing) 

Water 

F 

Gunnery 

ATC/ 

L 

10 

33 

20 

14 

21 

(depends  on 
requirements) 

AVTB 

S 

NBC  testing 

EPG 

L 

10 

33 

20 

14 

21 

S 

E3  testing 

Y  ship 
operations 

WSMR 

M 

60 

200 

120 

86 

128 

X 

X 

S 

E3  testing 
(limited) 

PAX 

15 

50 

30 

21 

32 

X 

X 

X 

67 


CTP 

Test 

Prerequis 

Test 

Priority 

Test  Days 

Pessi- 

Most 

Opti- 

Planning 

Best 

Medium 

Worst 

Area 

Plan/Sheet 

ite  test 

Venue 

of  Tests 

Required 

mistic 

Likely 

mistic 

Est 

S 

ROS  testing 

TBD 

L 

2 

7 

4 

3 

4 

X 

Other 

S 

(Signatures, 

TBD 

M 

25 

83 

50 

36 

53 

X 

etc.) 

c 

C4I  testing 

AVTB 

M 

0 

0 

0 

0 

X 

c 

RF  Testing 

FH 

L 

0 

0 

0 

0 

X 

c 

Interoperabilit 
y  testing 

Limited  Main 

AVTB 

M 

0 

0 

0 

0 

X 

F 

Gun  testing 

ATC 

8 

27 

16 

11 

17 

(Fly-Off) 

F 

COAX  testing 

ATC 

H 

2 

7 

4 

3 

4 

(VC  Firing) 
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The  first  cells  in  the  interior  of  Table  14  displays  the  functional  test  CTP  area:  LM 
=  land  mobility,  F  =  firepower,  S/HF  =  safety/human  factors,  WM  =  water  mobility,  C  = 
communications,  and  S  =  survivability. 

C.  TE  PLANNING  SCHEDULES 

The  TE  detailed  planning  numbers  are  assessed  as  to  whether  they  will  be 
tested  before  or  after  Milestone  C.  They  are  aggregated  into  total  numbers  by  CTP 
areas  for  each  test  schedule.  The  aggregated  schedule  includes  pessimistic,  optimistic, 
and  planning  estimate.  This  particular  example  is  for  before  MS-C.  The  TE  values  at  the 
start  of  the  schedule  development  are  provided  in  Table  15.  The  first  column  in  the 
interior  of  Table  15  displays  the  functional  test  CTP  area:  LM  =  land  mobility,  F  = 
firepower,  WM  =  water  mobility,  S/HF  =  safety/human  factors,  Surv  =  survivability,  and 
Comm  =  communications. 

Table  15.  TE  Aggregated  Estimate  (Days)  by  CTP  Area 


Test 

Pessimistic 

Planning 
Est.  ' 

Optimistic 

LM 

120 

77 

51 

F 

34 

22 

15 

WM 

67 

43 

29 

S/HF 

257 

164 

110 

Surv 

50 

32 

21 

Comm 

50 

32 

21 

There  are  three  schedules  that  are  produced  by  the  TE  planners,  which  are 
called  worse  case,  best  case  and  planning  estimate.  From  these,  the  final  schedule  is 
formed.  These  relate  to  the  pessimistic,  optimistic,  and  planning  estimate  columns, 
which  are  added  up  to  form  days  and  are  then  translated  into  months. 
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D.  WORST  CASE  (PESSIMISTIC)  SCHEDULE 


Table  16  contains  the  beginnings  of  the  worst  case  schedule.  The  last  column 
titled  “Actual”  is  an  adjustment  made  by  the  TE  planners  after  the  initial  monthly 
calculations  are  made. 


Table  1 6.  TE  Worst  Case  Calculations 


Days 

Months 

Actual 

LM 

120 

5.5 

5.5 

F 

34 

1.5 

1 

WM 

67 

3.0 

3 

S/HF 

257 

11.7 

11.5 

Surv 

50 

2.3 

3 

Comm 

50 

2.3 

3 

RGT 

0 

4 

26.3 

31.0 

Table  17  is  the  worst  case  schedule  generated  by  the  TE  planners,  and  is  based 
on  Table  16  using  the  “Actual”  column. 


Table  17.  TE  Monthly  Pessimistic  Schedule 


1 

2 

3 

4 

5 

6 

VI 

LM 

LM 

LM 

LM 

LM 

F 

V2 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF/LM 

V3 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

V4 

WM 

WM 

WM 

Comm 

Comm 

Comm 

V5 

Surv 

Surv 

Surv 

V6 

RGT1 

RGT1 

V7 

RGT1 

RGT1 

The  TE  planners  assign  locations  after  they  determine  the  test  makeup  and 
placement  on  the  schedule.  This  location  assignment  can  vary  based  on  available 
locations.  Figure  12  is  an  example  of  locations  that  TE  planners  would  potentially  assign 
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based  on  their  test  venue  assignment.  Figure  12  is  generated  for  comparison  to  the 
betajmax  schedule  generated  by  the  model. 


1 

2 

3 

4 

5 

6 

VI 

LM 

LM 

LM 

LM 

LM 

F 

V2 

SH/F 

SH/F 

SH/F 

SH/F 

SH/F 

SH/F  /  LM 

V3 

SHF 

SH/F 

SH/F 

SH/F 

SH/F 

SH/F 

V4 

WM 

WM 

WM 

Comm 

Comm 

Comm 

V5 

Surv 

Surv 

Surv 

V6 

RGT1 

RGT1 

V7 

RGT1 

RGT1 

Figure  12.  TE  Monthly  Pessimistic  Schedule  with  Test  Venue  Assignment 


E.  BEST  CASE  (OPTIMISTIC)  SCHEDULE 

Table  18  contains  the  beginnings  of  the  best  case  schedule.  The  last  column 
titled  “Actual”  is  an  adjustment  made  by  the  TE  planners  after  the  initial  monthly 
calculation  is  made. 


Table  18.  TE  Best  Case  Calculations 


Days 

Months 

Actual 

LM 

51 

2.3 

2.5 

F 

15 

0.7 

0.5 

WM 

29 

1.3 

2 

S/HF 

110 

5.0 

5 

Surv 

21 

1.0 

1 

Comm 

21 

1.0 

1 

RGT 

0 

4 

11.2 

16.0 

The  next  monthly  schedule  that  we  look  at  is  the  optimistic  schedule.  Table  19  is  the 
best  case  schedule  generated  by  the  TE  planners,  based  on  Table  18  using  the  “Actual” 
column. 
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Table  19.  TE  Monthly  Optimistic  Schedule 


1 

2 

3 

VI 

LM 

LM 

LM  /  F 

V2 

SH/F 

SH/F 

S/HF 

V3 

SH/F 

SH/F 

V4 

WM 

WM 

V5 

Surv 

Comm 

V6 

RGT1 

RGT1 

V7 

RGT1 

RGT1 

The  TE  planners  assign  locations  after  they  determine  the  test  makeup  and 
placement  on  the  schedule.  This  location  assignment  can  vary  based  on  available 
locations.  Figure  13  is  an  example  of  locations  that  TE  planners  would  assign  based  on 
their  test  venue  assignment.  Figure  13  is  generated  for  comparison  to  the  beta_min 
schedule  generated  by  the  model. 


1 

2 

3 

VI 

LM 

LM 

LM/F 

V2 

SH/F 

SH/F 

S/HF 

V3 

SH/F 

SH/F 

V4 

WM 

WM 

V5 

Surv 

Comm 

V6 

RGT1 

RGT1 

V7 

RGT1 

RGT1 

Figure  1 3.  TE  Monthly  Optimistic  Schedule  with  Test  Venue  Assignment 


F.  PLANNING  ESTIMATE  SCHEDULE 

Table  20  contains  the  beginnings  of  the  planning  estimate  schedule.  The  last 
column  titled  “Actual”  is  an  adjustment  made  by  the  TE  planners  after  the  initial  monthly 
calculation  is  made.  Note  that  RGT  is  planned  for  18  months,  but  only  seven  months  go 
on  this  planning  estimate  schedule  because  the  remainder  is  planned  on  the  schedule 
after  MS-C. 
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Table  20.  TE  Planning  Estimate  Calculations 


Days 

Months 

Actual 

LM 

77 

3.5 

3.5 

F 

22 

1.0 

1 

WM 

43 

2.0 

2 

S/HF 

164 

7.5 

8.5 

Surv 

32 

1.5 

2 

Comm 

32 

1.5 

1 

RGT 

0 

18.0 

7 

34.8 

25.0 

The  next  monthly  schedule  that  we  look  at  is  the  planning  estimate  schedule. 
Table  21  is  the  planning  estimate  schedule  generated  by  the  TE  planners  and  is  based 
on  Table  20  using  the  “Actual”  column. 


Table  21 .  TE  Monthly  Planning  Estimate  Schedule 


. 

1 

2 

3 

4 

5 

VI 

LM 

LM 

SH/F 

SH/F 

SH/F 

V2 

SH/F 

LM 

SH/F/LM 

SH/F 

SH/F 

V3 

SH/F 

SH/F 

F 

V4 

WM 

WM 

RGT 

V5 

Surv 

Surv 

Comm 

V6 

RGT1 

RGT1 

RGT2 

V7 

RGT1 

RGT1 

RGT2 

The  TE  planners  assign  locations  after  they  determine  the  test  makeup  and 
placement  on  the  schedule.  This  location  assignment  can  vary  based  on  available 
locations.  Figure  14  is  an  example  of  locations  that  TE  planners  would  assign  based  on 
their  test  venue  assignment.  Figure  14  is  generated  for  comparison  to  the  beta_mean 
schedule  generated  by  the  model. 
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1 

2 

3 

4 

5 

VI 

LM 

LM 

SH/F 

SH/F 

SH/F 

V2 

SH/F 

LM 

SH/F  /  LM 

SH/F 

SH/F 

V3 

SH/F 

SH/F 

F 

V4 

WM 

WM 

RGT 

V5 

Surv 

Surv 

Comm 

V6 

RGT1 

RGT1 

RGT2 

V7 

RGT1 

RGT1 

RGT2 

Figure  14.  TE  Monthly  Planning  Estimate  Schedule  with  Test  Venue 

Assignment 


G.  MODEL  SCHEDULES 

The  five  schedules  that  are  generated  by  the  model  are  based  on  a  Monte  Carlo 
beta  distribution  and  consist  of  beta_max,  beta_min,  beta_mode,  and  beta_mean 
schedules.  The  t_periods  model  schedule  is  the  100%  availability  schedule,  which  is 
shorter  than  betajmin.  The  three  schedules  that  are  provided  by  TE  are  based  on  a 
Program  Evaluation  and  Review  Technique  (PERT)  beta  distribution  of  pessimistic, 
optimistic  and  planning  estimate.  The  beta_max  model  schedule  equates  to  the  TE 
planner’s  pessimistic  schedule,  and  the  beta_min  model  schedule  equates  to  the  TE 
planner’s  optimistic  schedule  (Edwards  2015,  6).  The  betajmean  model  schedule 
equates  to  the  TE  planner’s  planning  estimate  schedule.  The  betajmode  schedule 
equates  to  the  TE  planner’s  most  likely  column  from  Table  14.  The  t_periods  model 
schedule  equates  to  the  TE  planner’s  “Test  Days  Required”  100%  availability  column 
from  Table  14. 

An  aggregation  of  numbers  is  performed  against  the  model  run  data  that  is 
similar  in  nature  to  the  TE  planner’s  efforts.  Table  22  provides  the  number  of  days  by 
CTP  area  for  each  model  run  schedule. 
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Table  22.  TE  Aggregated  Schedule  (Days)  by  CTP  Area  for  the  Model  Run 


Test 

Beta max 

Beta mean 

Beta mode 

Beta min 

t periods 

LM 

128 

81 

76 

55 

38 

F 

34 

23 

22 

15 

15 

WM 

91 

57 

54 

37 

27 

S/HF 

582 

394 

376 

279 

195 

Surv 

50 

32 

30 

21 

15 

Comm 

50 

33 

32 

21 

21 

RGT1 

88 

88 

88 

88 

88 

RGT2 

66 

66 

66 

66 

66 

The  columns  that  are  most  similar  to  the  set  of  three  schedules  that  are  used  by 
the  TE  planners  are  indicated  in  the  Table  22  by  bold  text.  Adding  in  the  TE  planning 
schedule  allows  comparison  and  is  provided  in  Table  23. 

TE  schedule  information  contained  in  the  tables  provided  in  this  section  is  based 
on  data  provided  by  the  PM  AAA  TE  group  (Ferguson,  Louis  R.  and  Albert  B. 
Hanneman,  email  message  to  author,  April  1, 2015). 
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Table  23.  TE  Aggregated  Schedule  (Days)  by  CTP  Area  and  Model  Run  Data  Compared 


Test 

T&E 

Pessimistic 

Model 

Beta_max 

TE 

Planning 

Estimate 

Model 

Beta_mean 

TE 

Most 

Likely 

Model 

Beta_mode 

TE 

Optimistic 

Model 

Beta_min 

Model 

t_periods 

LM 

120 

128 

77 

81 

73 

76 

51 

55 

38 

F 

34 

34 

22 

23 

21 

22 

15 

15 

15 

WM 

67 

91 

43 

57 

41 

54 

29 

37 

27 

S/HF 

257 

582 

164 

394 

154 

376 

110 

279 

195 

Surv 

50 

50 

32 

32 

30 

30 

21 

21 

15 

Comm 

50 

50 

32 

33 

30 

32 

21 

21 

21 

RGT1 

40 

88 

80 

88 

80 

88 

40 

88 

88 

RGT2 

40 

66 

60 

66 

60 

66 

40 

66 

66 
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As  can  be  seen  in  Table  23,  the  data  that  is  used  as  input  for  the  model 
run  is  similar,  but  does  not  exactly  match,  the  TE  planner’s  data.  Of  particular 
difference  are  the  S/HF  numbers,  because  the  S/HF  numbers  used  in  the  model 
run  include  the  initial  inspection  and  safety  checkout  test  event,  which  is  not 
explicitly  called  out  in  the  TE  planner’s  schedule.  The  numbers  for  F  and  Comm 
do  not  show  up  on  the  TE  detailed  spreadsheet  given  in  Table  14  with  numbers 
and  “X”s  in  the  three  schedule  columns.  Comm  and  F  are  added  as  an  aggregate 
number  to  each  of  the  TE  schedules  after  the  initial  detailed  spreadsheet 
assessment.  Therefore,  in  the  model  run,  they  are  added  to  the  input  files  as 
single  test  events  of  Comm  and  F. 

RGT  is  not  specifically  listed  by  the  TE  planners  in  Table  14.  RGT  is 
something  that  is  known  and  is  normally  added  to  the  schedule  after  the  rest  of 
the  test  events  are  on  the  schedule.  RGT  is  an  iterative  process.  RGT  starts  with 
RGT1.  RGT1  informs  the  PMO  of  the  system  reliability.  Reliability  improvements 
are  incorporated  through  a  CAP1 ,  which  is  then  tested  as  part  of  RGT2.  RGT2  is 
then  incorporated  as  part  of  CAP2  and  is  ultimately  tested  in  Reliability 
Qualification  Testing  (RQT)  to  show  the  system  reliability  growth  for  production. 
RGT  is  incorporated  into  the  model  run  by  setting  up  seven  separate  test  events. 
Four  RGT  test  events  are  high  priority  and  three  RGT  test  events  are  medium 
priority.  The  four  high  priority  RGT  test  events  equate  to  RGT1  and  the  three 
medium  priority  test  events  equate  to  RGT2.  Calculations  for  this  particular  test 
schedule  indicate  that  18  months  of  RGT  are  needed.  The  remaining  eleven 
RGT  months  are  added  to  the  overall  schedule  after  the  milestone  C  event, 
which  are  tied  to  the  end  of  the  priority  testing  (after  this  schedule  is  complete). 

In  order  to  compare  the  model  run  with  the  TE  planner  schedules,  the 

detailed  daily  schedules  generated  and  shown  in  Appendix  B  are  translated  into 

monthly  schedules.  The  daily  schedules  are  manually  generated  from  the  model 

run,  aggregating  20  days  of  the  detailed  daily  schedules  into  single  blocks  on  a 

MS  Excel  schedule.  The  monthly  schedules  are  similar  to  what  the  TE  planners 

use  during  the  planning  process.  The  aggregated  monthly  schedules  are  similar 
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to  the  detailed  daily  schedules  in  Appendix  B  in  structure  and  colored  cells, 
except  the  cells  do  not  contain  the  test  event  identification  numbers.  The  CTP 
area  is  used  to  aggregate  the  test  events  to  a  higher  level. 

H.  COMPARISON  OF  SCHEDULES:  TE  PESSIMISTIC  VERSUS 

BET AM AX 

Figure  15  is  generated  manually  based  on  the  detailed  daily  schedules 
given  in  Appendix  B,  Section  A  (Beta_max),  aggregating  them  to  a  monthly 
schedule  based  on  a  20  day  test-month.  This  schedule  formatting  is  performed  for 
comparison  to  make  the  schedule  appear  in  the  same  format  as  if  the  TE  planners 
had  developed  it. 
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Figure  15.  Model  Beta_max  Monthly  Schedule 


The  betajmax  monthly  schedule  is  aggregated  from  the  detailed  daily 
schedule,  with  test  assets  1  through  7  on  the  y-axis  (TA1  -  TA7)  and  monthly  test 
periods  of  month  1  through  8  on  the  x-axis  (Ml  -  M8).  The  interior  cells  contain  the 
test  venue,  indicated  by  cell  color,  and  the  aggregated  CTP  area. 

As  an  example,  test  asset  3  starts  out  at  ATC  (green)  in  month  1  performing 
S/HF  testing,  which  continues  through  month  2,  and  partially  into  month  3.  In  month 
3,  TA3  then  moves  into  Reliability  Growth  Testing  1  (RGT1)  testing,  which  continues 
into  month  4.  TA3  then  moves  into  LM  testing  in  month  4,  which  continues  into 
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month  5  and  month  6.  In  month  7,  TA3  then  moves  into  RGT2,  which  continues  into 
month  8. 

Before  looking  at  the  details  of  the  differences  between  the  Figure  12  and 
Figure  15  schedules,  the  differences  between  TE  planner  input  and  model  input 
need  to  be  understood.  Table  24  shows  that  there  is  a  length  difference  between 
the  schedules  of  three  months  due  to  the  difference  in  input  data,  mostly  due  to 
the  added  S/HF.  Therefore,  we  cannot  expect  an  exact  match  between  the  TE 
pessimistic  schedule  and  the  betajmax  model  schedule. 


Table  24.  TE  versus  Model  Worst  Case  Calculations 


TE  Pessimistic 

Beta max  Model 

Test 

Days 

Months 

Actual 

Days 

Months 

Delta 

LM 

120 

5.5 

5.5 

128 

6.4 

0.9 

F 

34 

1.5 

1 

34 

1.7 

0.7 

WM 

67 

3 

3 

91 

4.6 

1.6 

S/HF 

257 

11.7 

11.5 

582 

29.1 

17.6 

Surv. 

50 

2.3 

3 

50 

2.5 

-0.5 

Comm 

50 

2.3 

3 

50 

2.5 

-0.5 

RGT 

0 

4 

88 

7 

3 

26.3 

31 

53.8 

22.8 

Comparing  Figure  12  and  Figure  15  schedules  side  by  side,  the  TE 
pessimistic  schedule  and  the  beta_max  model  both  have  four  test  assets  at  ATC, 
one  test  asset  at  WSMR,  and  two  test  assets  at  AVTB  (ignoring  the  first  month  of 
S/HF  testing  on  the  model  schedule).  The  majority  of  testing  is  performed  at  ATC 
on  both  schedules.  Land  mobility,  water  mobility,  survivability,  and  RGT1  CTP 
areas  are  first  on  both  schedules  (ignoring  the  first  month  of  S/HF  testing  on  the 
model  schedule).  The  Comm  CTP  area  is  last  on  both  schedules.  Firepower 
testing  happens  in  the  middle  for  the  model  schedule  and  at  the  end  of  the  TE 
planning  schedule.  RGT2  is  at  the  end  of  the  model  schedule  and  does  not  exist 
on  the  TE  schedule,  which  is  due  to  a  difference  in  the  input  data.  The  model 
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choses  to  go  to  YPG  instead  of  ATC  at  the  end  for  S/HF  t41  Climatic  Chambers, 
which  is  a  medium  priority  task,  whereas  the  TE  planner’s  schedule  uses  ATC. 
The  choice  of  YPG  makes  sense  since  YPG  is  closer  to  AVTB  than  ATC  is. 
However,  it  would  have  made  more  sense  to  leave  TA6  at  AVTB  and  to  leave 
TA1  at  ATC  instead  of  moving  to  TA1  to  AVTB  and  TA6  to  YPG.  Additionally,  the 
beta_max  schedule  and  the  detailed  schedules  in  Appendix  B,  Section  B 
(Beta_max)  have  some  blank  space  in  the  middle,  which  should  not  be  there. 

When  looking  at  these  two  schedules,  it  is  apparent  that  the  model  has 
more  overlap  of  CTP  areas  than  the  TE  planner’s  schedule.  The  model  CTP  area 
overlap  is  expected  since  in  the  model  generated  the  schedule  based  on  the 
detailed  test  events  and  based  on  priorities  and  precedence.  The  manual 
aggregation  to  CTP  area  happens  after  the  model  test  schedule  is  generated. 
The  TE  planners  add  the  CTP  areas  together  before  the  schedule  is  generated. 
As  expected,  the  model  input  does  not  completely  match  what  is  used  for  the  TE 
planners,  which  makes  the  betajmax  schedule  three  months  longer  than  the  TE 
pessimistic  schedule.  In  spite  of  these  differences,  overall,  the  model  betajmax 
schedule  is  a  reasonable  schedule  option. 

I.  COMPARISON  OF  SCHEDULES:  TE  OPTIMISTIC  VERSUS  BETA  MIN 

Figure  16  is  generated  manually  based  on  the  detailed  daily  schedules  in 
Appendix  B,  Section  C  (Betajmin),  aggregating  them  to  a  monthly  schedule 
based  on  a  20  day  test-month.  This  schedule  formatting  is  performed  for 
comparison  to  make  the  schedule  appear  in  the  same  format  as  if  the  TE  planners 
had  developed  it. 
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Figure  16.  Model  Betajmin  Monthly  Schedule 


The  betajnin  monthly  schedule  is  aggregated  from  the  Appendix  B  daily 
schedules,  with  test  assets  TA1  -  TA7  on  the  y-axis  and  monthly  test  periods  Ml 
-  M4  on  the  x-axis.  The  interior  cells  contain  test  venue,  indicated  by  cell  color, 
and  the  aggregated  CTP  areas. 

As  an  example,  TA3  starts  out  at  ATC  (green)  in  month  1,  performing 
S/HF  testing.  TA3  remains  at  ATC  and  moves  into  RGT1  in  month  2.  TA3  moves 
into  S/HF  testing  in  month  3.  TA3  then  moves  into  LM  testing  in  month  4. 

Before  looking  at  the  details  of  the  differences  between  the  Figure  13  and 
the  Figure  16  schedules,  the  differences  between  TE  planner  input  and  model 
input  need  to  be  understood.  Table  25  shows  that  there  is  a  length  difference 
between  the  schedules  of  two  months  due  to  the  difference  in  input  data,  mostly 
due  to  the  added  S/HF.  Therefore,  we  cannot  expect  an  exact  match  between 
the  TE  optimistic  schedule  and  the  betajmin  model  schedule. 
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Table  25.  TE  Optimistic  versus  Model  Beta_min  Calculations 


Comparing  Figure  13  and  Figure  16  side  by  side,  the  TE  optimistic 
schedule  and  the  beta_min  model  schedule  both  start  with  four  test  assets  at 
ATC,  one  test  asset  at  WSMR,  and  two  test  assets  at  AVTB  (ignoring  the  first 
month  of  S/HF  testing  on  the  model  schedule).  Land  mobility,  water  mobility, 
survivability,  and  RGT1  CTP  areas  are  first  on  both  schedules  (ignoring  the  first 
month  of  S/HF  testing  on  the  model  schedule).  The  Comm  CTP  area  is  last  on 
both  schedules.  Firepower  testing  happens  in  the  middle  for  the  model  schedule 
and  at  the  end  of  the  TE  planning  schedule.  RGT2  testing  is  at  the  end  of  the 
model  schedule  and  does  not  exist  on  the  TE  schedule,  which  is  due  to  a 
difference  in  the  input.  The  betajmin  schedule  has  some  blank  spaces  here  and 
there  in  the  middle  of  the  detailed  schedules  in  Appendix  B,  Section  D 
(Betajmin),  which  should  not  be  there 

When  looking  at  these  two  schedules,  it  is  apparent  that  the  model  has 

more  overlap  of  CTP  areas  than  the  TE  planner’s  schedule.  The  model  CTP  area 

overlap  is  expected  because  the  model  generated  the  schedule  based  on  the 

detailed  test  events  and  based  on  priorities  and  precedence.  The  manual 

aggregation  to  CTP  area  happens  after  the  model  test  schedule  is  generated. 

The  TE  planners  add  the  CTP  areas  together  before  the  schedule  is  generated. 

As  expected,  the  model  input  does  not  completely  match  what  is  used  for  the  TE 

planners,  which  makes  the  betajmin  schedule  two  months  longer  than  the  TE 
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optimistic  schedule.  The  model  schedule  is  more  accurate  relative  to  test  event 
priorities  and  places  the  medium  priority  tasks  at  the  end.  This  approach  is  in 
conflict  with  the  approach  of  aggregating  to  CTP  area  before  putting  it  on  the 
schedule  as  the  TE  planners  do.  In  spite  of  these  differences,  overall,  the  model 
betajmin  schedule  is  a  reasonable  schedule  option. 

J.  COMPARISON  OF  SCHEDULES:  TE  PLANNING  ESTIMATE  VERSUS 

BETAMEAN 

Figure  17  is  generated  manually  based  on  the  detailed  daily  schedules  in 
Appendix  B,  Section  E  (Beta_mean),  aggregating  them  to  a  monthly  schedule 
based  on  a  20  day  test-month.  This  schedule  formatting  is  performed  for 
comparison  to  make  the  schedule  appear  in  the  same  format  as  if  the  TE  planners 
had  developed  it. 
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Figure  17.  Model  Beta_mean  Monthly  Schedule 


The  betajmean  monthly  schedule  is  aggregated  from  the  daily  Appendix 
B  schedule,  with  test  assets  TA1  -  TA7  on  the  y-axis  and  monthly  test  periods 
Ml  -  M6  on  the  x-axis.  The  interior  cells  contain  test  venue,  indicated  by  cell 
color,  and  the  aggregated  CTP  areas. 

As  an  example,  test  asset  2  starts  out  at  ATC  (green)  in  month  1, 
performing  S/HF  testing.  TA2  remains  at  ATC  in  month  2,  continues  S/HF 
testing,  and  transitions  into  F  testing  at  the  end  of  month  2.  In  month  3,  TA2 
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continues  F  testing  and  transitions  into  S/HF  testing.  TA2  continues  S/HF  testing 
for  months  4,  5,  and  6. 

Before  looking  at  the  details  of  the  differences  between  the  Figure  14  and 
Figure  17  schedules,  the  differences  between  TE  planner  input  and  model  input 
need  to  be  understood.  Table  26  shows  that  there  is  a  length  difference  of  two 
months  due  to  the  difference  in  input  data.  Therefore,  we  cannot  expect  an  exact 
match  between  the  TE  planning  estimate  schedule  and  the  betajmean  model 
schedule. 


Table  26.  TE  Planning  Estimate  versus  Model  Betajmean  Calculations 


Comparing  Figure  14  and  Figure  17  side  by  side,  the  TE  schedule  has 
three  test  assets  at  ATC,  whereas  the  betajmean  has  four  test  assets  at  ATC. 
The  TE  schedule  has  three  test  assets  at  AVTB,  whereas  the  betajmean  has 
two  test  assets  at  AVTB.  Both  schedules  have  one  test  asset  at  WSMR.  Land 
mobility,  water  mobility,  S/HF,  survivability,  and  RGT1  are  first  (after  the  S/HF 
that  is  added  at  the  beginning  of  the  beta-mean  schedule).  The  Comm  CTP  area 
is  last  on  both  schedules.  The  betajmean  schedule  does  move  a  test  asset  to 
YTC  after  WSMR.  This  test  venue  movement  makes  sense  because  YTC  is 
closer  than  ATC.  In  contrast,  a  test  planner  might  chose  ATC  or  AVTB  instead 
because  vehicles  are  then  at  less  test  venues. 
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When  looking  at  these  two  schedules,  it  is  apparent  that  the  model  has 
more  overlap  of  CTP  areas  than  the  TE  planner’s  schedule.  The  model  CTP  area 
overlap  is  expected  because  the  model  generated  the  schedule  based  on  the 
detailed  test  events  and  based  on  priorities  and  precedence.  The  manual 
aggregation  to  CTP  area  happens  after  the  model  test  schedule  is  generated. 
The  TE  planners  add  the  CTP  areas  together  before  the  schedule  is  generated. 
As  expected,  the  model  input  does  not  completely  match  what  is  used  for  the  TE 
planners,  which  makes  the  betajmean  schedule  two  months  longer  than  the  TE 
planning  estimate  schedule.  The  model  schedule  is  more  accurate  relative  to  test 
event  priorities  and  places  the  medium  priority  tasks  at  the  end.  This  approach  is 
in  conflict  with  the  approach  of  aggregating  to  CTP  area  before  putting  it  on  the 
schedule  as  the  TE  planners  do.  In  spite  of  these  differences,  overall,  the  model 
betajmean  schedule  is  a  reasonable  schedule  option. 

K.  COMPARISON  OF  SCHEDULES:  TE  PLANNING  ESTIMATE  VERSUS 

BETAMODE 

Figure  18  is  generated  manually  based  on  the  detailed  schedules  in 
Appendix  B,  Section  D  (Betajmode),  aggregating  them  to  a  monthly  schedule 
based  on  a  20  day  test-month.  This  schedule  formatting  is  performed  to  make  the 
schedule  appear  in  the  same  format  as  if  the  TE  planners  had  developed  it. 
Betajmode  testing  equates  to  the  most  likely  testing  column  in  Table  15. 
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Figure  18.  Model  Betajmode  Monthly  Schedule 
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The  beta_mode  monthly  schedule  is  aggregated  from  the  daily  test 
periods  schedule,  with  test  assets  TA1  -  TA7  on  the  y-axis  and  monthly  test 
periods  Ml  -  M6  on  the  x-axis.  The  interior  cells  contain  test  venue,  indicated  by 
cell  color  (AVTB  =  blue,  ATC  =  green,  YPG  =  purple,  and  WSMR  =  salmon),  and 
the  aggregated  CTP  areas.  For  example,  TA5  starts  out  at  WSMR  (salmon)  in 
month  1,  performing  S/HF  testing.  In  month  2,  TA5  remains  at  WSMR  and  moves 
into  Survivability  (Surv)  testing  in  month  2,  which  continues  into  month  3. 
Partway  through  month  3,  TA5  moves  into  S/HF  testing.  In  month  4,  TA5  moves 
to  YPG  (purple)  and  performs  S/HF  testing  into  month  5,  where  TA5  testing  is 
complete. 

Before  looking  at  the  details  of  the  differences  between  the  Figure  14  and 
Figure  18  schedules,  the  differences  between  TE  planner  input  and  model  input 
need  to  be  understood.  Table  27  shows  that  there  is  a  length  difference  of  two 
months  due  to  the  difference  in  input  data.  Therefore,  we  cannot  expect  an  exact 
match  between  the  TE  planning  estimate  schedule  and  the  beta_mode  model 
schedule. 


Table  27.  TE  Planning  Estimate  versus  Model  Betajmode  Calculations 


While  betajmode  is  not  expected  to  exactly  track  to  the  TE  planning 
estimate  schedule,  they  are  the  closest  for  comparison  purposes.  When 
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comparing  Figure  14  to  Figure  18,  as  expected,  the  model  input  does  not 
completely  match  what  is  used  by  the  TE  planners,  and  is  longer.  Therefore,  we 
cannot  expect  an  exact  match  between  the  two  schedules.  Month  6  has  so  few 
entries,  that  we  could  actually  delete  month  6  from  the  betajmode  schedule. 

Comparing  Figure  14  and  Figure  18  side  by  side,  both  schedules  have 
three  test  assets  at  ATC,  three  test  assets  at  AVTB,  and  one  test  asset  at  WSMR 
(ignoring  the  first  month  of  S/HF  testing  on  the  model  schedule).  Land  mobility, 
water  mobility,  S/HF,  survivability,  and  RGT1  are  first  (ignoring  the  first  month  of 
S/HF  testing  on  the  model  schedule).  The  Comm  CTP  area  is  last  on  both 
schedules.  The  betajmode  schedule  does  move  a  test  asset  to  YTC  after 
WSMR.  This  test  venue  movement  makes  sense  because  it  is  closer  than  ATC, 
whereas  a  test  planner  might  chose  ATC  or  AVTB  instead  because  vehicles  are 
then  at  less  test  venues. 

When  looking  at  these  two  schedules,  it  is  apparent  that  the  model  has 
more  overlap  of  CTP  areas  than  the  TE  planner’s  schedule.  The  model  CTP  area 
overlap  is  expected  because  the  model  generated  the  schedule  based  on  the 
detailed  test  events  based  on  priorities  and  precedence.  The  manual  aggregation 
to  CTP  area  happens  after  the  test  schedule  is  generated.  The  TE  planners  add 
the  CTP  areas  together  before  the  schedule  is  generated.  As  expected,  the 
model  input  does  not  completely  match  what  is  used  for  the  TE  planners,  which 
makes  the  betajmode  schedule  two  months  longer  than  the  TE  optimistic 
schedule.  The  model  schedule  is  more  accurate  relative  to  test  event  priorities 
and  places  the  medium  priority  tasks  at  the  end.  This  approach  is  in  conflict  with 
the  approach  of  aggregating  to  CTP  area  before  putting  on  the  schedule,  as  the 
TE  planners  do.  In  spite  of  these  differences,  overall,  the  model  betajmode 
schedule  is  a  reasonable  schedule  option. 
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L.  COMPARISON  OF  SCHEDULES  :  TE  PLANNING  ESTIMATE  VERSUS 
T_  PERIODS 

Figure  19  is  generated  manually  based  on  the  detailed  schedules  in 
Appendix  B,  Section  B  (t_periods),  aggregating  them  to  a  monthly  schedule 
based  on  a  20  day  test-month.  This  schedule  formatting  is  performed  to  make  the 
schedule  appear  in  the  same  format  as  if  the  TE  planners  had  developed  it.  Testing 
schedule  t_periods  equates  to  the  test  days  required  column  in  Table  14. 
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Figure  19.  Model  t_periods  Monthly  Schedule 


This  monthly  schedule  is  aggregated  from  the  daily  test  periods  schedule, 
with  test  assets  TA1  -  TA7  on  the  y-axis  and  monthly  test  periods  Ml  -  M4  on  the 
x-axis.  The  interior  cells  contain  test  venues  indicated  by  cell  color  and  the 
aggregated  CTP  areas.  For  example,  TA6  starts  out  at  AVTB  (blue)  in  month  1, 
performing  S/HF  testing.  TA6  remains  at  AVTB  and  moves  into  water  mobility 
testing  in  month  2  but  moves  into  RGT1  partway  through  month  2.  TA6  continues 
RGT1  testing  in  month  3  and  transitions  into  Comm  testing  in  month  3, 
continuing  through  month  4. 

Before  looking  at  the  details  of  the  differences  in  the  actual  schedules,  the 
differences  between  TE  planner  input  and  model  input  need  to  be  understood. 
Table  28  shows  the  input  data  differences  between  Figure  14  and  Figure  19. 
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Table  28  shows  that  there  is  little  overall  length  difference  between  the  planning 
estimate  and  the  t_periods  due  to  the  difference  in  input  data,  although  there  are 
specific  differences  in  CTP  area  numbers.  For  example,  LM  CTP  area  is  3.5 
months  for  the  TE  planning  estimate  schedule,  whereas  the  t_periods  LM  CTP 
area  is  1 .9  months,  which  gives  a  delta  of  1 .6  months. 

Table  28.  TE  Planning  Estimate  versus  Model  t_periods  Calculations 


CTP 

Area 

TE  Planning  Estimate 

t_periods  Model 

Days 

Months 

Actual 

Days 

Months 

Delta 

LM 

77 

3.5 

3.5 

38 

1.9 

1.6 

F 

22 

1.0 

1 

15 

0.8 

0.3 

WM 

43 

2.0 

2 

27 

1.4 

0.7 

S/HF 

164 

7.5 

8.5 

195 

9.8 

-1.3 

Surv. 

32 

1.5 

2 

15 

0.8 

1.3 

Comm 

32 

1.5 

1 

21 

1.1 

-0.1 

RGT 

0 

18.0 

7 

154 

7.7 

-3.7 

34.8 

25.0 

23.5 

-1.3 

Although  the  TE  planning  estimate  schedule  and  the  t_periods  model 
schedule  have  the  closest  schedule  data  for  comparison  purposes,  they  are  not 
expected  to  track  to  each  other  exactly.  Therefore,  when  comparing  Figure  14  to 
Figure  19,  we  do  not  expect  an  exact  match  between  the  two  schedules. 

Comparing  Figure  14  and  Figure  19  side  by  side,  the  TE  schedule  initially 
has  three  test  assets  at  ATC,  one  test  asset  at  WSMR,  and  three  test  assets  at 
AVTB,  whereas  the  t_periods  model  schedule  initially  has  four  test  assets  at  ATC 
and  AVTB  and  one  at  WSMR.  In  the  second  month,  however,  the  schedules 
track  to  the  same  locations.  There  is  a  starting  location  that  is  a  model  input  that 
caused  this  initial  location  difference.  Additionally,  land  mobility,  water  mobility, 
survivability  and  RGT1  are  the  first  CTP  areas  (after  the  S/HF  that  is  added  at 
the  beginning  of  the  t_periods  schedule).  The  Comm  CTP  area  is  last  on  both 
schedules.  RGT1  occurs  at  the  beginning  of  the  schedule,  and  RGT2  occurs  at 
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the  end  of  both  schedules.  One  difference  between  the  two  schedules  is  F  is 
earlier  on  the  model  schedule  and  later  on  the  TE  schedule. 

When  looking  at  these  two  schedules,  it  is  apparent  that  the  model  has 
more  overlap  of  CTP  areas  than  the  TE  planner’s  schedule.  The  model  CTP  area 
overlap  is  expected  because  in  the  model  generated  the  schedule  based  on  the 
detailed  test  events  and  based  on  priorities  and  precedence.  The  manual 
aggregation  to  CTP  area  happens  after  the  model  test  schedule  is  generated. 
The  TE  planners  add  the  CTP  areas  together  before  the  schedule  is  generated. 
As  expected,  the  model  input  does  not  completely  match  what  is  used  for  the  TE 
planners,  although  it  is  close.  The  model  schedule  is  more  accurate  relative  to 
test  event  priorities  and  places  the  medium  priority  tasks  at  the  end.  This 
approach  is  in  conflict  with  the  approach  of  aggregating  to  CTP  area  before 
putting  it  on  the  schedule  as  the  TE  planners  do.  In  spite  of  these  differences, 
overall,  the  model  t_periods  schedule  is  a  reasonable  schedule  option. 

The  actual  schedule  used  by  TE  for  this  program  is  given  in  Figure  20. 
The  Figure  20  schedule  tracks  to  the  planning  estimate  schedule,  with  insertion 
of  the  CAP  period,  the  OA,  and  the  additional  eleven  months  of  RGT. 
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Figure  20.  TE  Planning  Schedule  Used 
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M.  MODEL  VALIDATION  SYNOPSIS 


By  looking  at  all  of  the  schedules  and  the  comparison  data,  the  model  is 
assessed  against  the  Table  1  operational  needs  from  a  validation  perspective  in 
Table  29.  When  assessing  from  this  perspective,  the  only  options  are  MET,  NOT 
MET,  and  NOT  VALIDATED. 
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Table  29.  Operational  Needs  and  Model  Verification  Results 


Need  # 

Title 

Tier 

Operational  Needs 

Model 

Verification 

Assessment 

Model  Validation  Assessment 

N1 

Multiple 

Asset 

Types  (0) 

A  PA 

The  TE  RSCSP  model  low 
priority  test  events  shall  be 
allowed  to  go  beyond  the 
test  period. 

NOT  VERIFIED 

NOT  VALIDATED.  Test  data  is 
insufficient  to  answer  the 
question,  although  the  model  has 
the  ability.  Additional  model  runs 
and  verification  efforts  would  be 
beneficial. 

N2 

Multiple 

Assets 

(T=0) 

KSA 

The  TE  RSCSP  model 
shall  allow  multiple  test 
assets  to  be  tested 
simultaneously. 

PARTIALLY  MET 

MET.  While  multiple  test  assets 
are  tested  simultaneously, 
multiple  test  asset  types  (0)  have 
not  been  verified.  Additional 
model  runs  and  verification  efforts 
would  be  beneficial. 

N3 

Time 

Period 

Asset 

Availability 

(0) 

A  PA 

The  TE  RSCSP  model 
shall  have  a  time  period 
test  asset  availability 
constraint. 

NOT  VERIFIED 

NOT  VALIDATED.  Test  data 
insufficient  to  answer  the 
question,  although  the  model  has 
the  ability.  Additional  model  runs 
and  verification  efforts  would  be 
beneficial. 

N4 

Venue 

Distance 

(T=0) 

KSA 

The  TE  RSCSP  model 
shall  minimize  test 
movement  between  test 

venues. 

MET 

MET.  The  model  minimized 
movement  between  test  venues. 
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Need  # 

Title 

Tier 

Operational  Needs 

N5 

Event 

Priority 

Placement 

(T=0) 

KSA 

The  TE  RSCSP  model 
shall  place  test  events 
based  on  priority. 

N6 

Multiple 

Venues 

(T=0) 

A  PA 

The  TE  RSCSP  model 
shall  allow  multiple  test 
venues  to  be  used  at  the 
same  time. 

N7 

Venue 
Choice  by 
Event 
(T=0) 

A  PA 

The  TE  RSCSP  model 
shall  allow  test  venue 
choice  by  test  event. 

N8 

Multiple 

Events 

(T=0) 

A  PA 

The  TE  RSCSP  model 
shall  allow  multiple  test 
events  to  occur  at  the 
same  time. 

N9 

Precedenc 

e 

Relationshi 

P  (T=0) 

A  PA 

The  TE  RSCSP  model 
shall  have  test  event 
predecessor  and 
successor  relationship 
constraints. 

N10 

Deadline 
for  each 
Priority 
(T=0) 

A  PA 

The  TE  RSCSP  model  shall 
place  test  assets  based  on 
a  deadline  for  each  priority 
type. 
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Model 

Verification 

Assessment 


Model  Validation  Assessment 


PARTIALLY  MET 

MET.  The  model  is  doing  a  good 
job  of  placing  test  events  based 
on  priority,  Additional  model  runs 
and  verification  efforts  would  be 
beneficial. 

MET 

MET.  Multiple  test  venues  are 
used  at  the  same  time. 

MET 

MET.  Test  venues  are  chosen  by 
test  event,  with  multiple  options 
available. 

MET 

MET.  The  test  schedules  show 
that  multiple  test  events  happen  at 
the  same  time,  and  the  model 
allows  test  events  to  occur  on 

MET 

multiple  test  assets 
simultaneously. 

MET.  Test  event  predecessor  and 
successor  relationships  are 
successful. 

PARTIALLY  MET 

MET.  Additional  model  runs  and 
verification  efforts  would  be 
beneficial  for  deadlines  not 
currently  verified. 

Need  #  Title 


Tier  Operational  Needs 


KPP  The  TE  RSCSP  model  shall 
minimize  the  time  period 
used  for  test  events. 


APA  The  TE  RSCSP  model  shall 
allow  TE  personnel  to  input 
information  in  MS  Excel. 


KPP  The  TE  RSCSP  model  shall 
output  schedules  in  a  MS 
Excel  Format  showing  each 
vehicle,  using  colors  for  test 
site,  and  using  blocks  that 
aggregate  the  information 
into  the  month. 

APA  The  TE  RSCSP  model  shall 
use  MS  Excel  only.  This 
means  that  a  specialized 
tool  will  not  be  used  for  the 
model. 


94 


Model  Model  Validation  Assessment 

Verification 

Assessment 


MET  MET.  Additional  model  runs  and 

verification  efforts  would  be 
beneficial.  While  there  are  some 
blanks  in  the  betajmax  schedule, 
the  blanks  are  believed  to  be  due 
to  the  shortening  of  the  time  to  run 
the  model. 

PARTIALLY  MET  MET.  While  the  current  approach 

is  not  ideal,  the  approach  is 
acceptable.  The  MS  Excel  input 
approach  is  an  area  that  can  be 
improved  upon. 

NOT  MET  NOT  MET.  This  is  an  area  that 

can  be  improved  upon. 


NOT  MET 


NOT  MET.  The  use  of  a 
specialized  tool  is  an  area  that 
can  be  improved  upon. 


OT  validation  assessment  of  the  capabilities  results  follow: 

•  One  of  two  KPP  is  assessed  as  MET. 

•  Three  of  three  KSAs  are  assessed  as  MET. 

•  Six  of  six  Threshold  =  Objective  APAs  are  assessed  as  MET. 

•  Zero  of  three  Objective  APAs  are  assessed  as  MET 

The  majority  of  the  capabilities  requested  for  the  model  are  assessed  as 
MET.  Discussions  follow  of  the  ones  that  are  not  assessed  as  MET: 

The  “MS  Excel  Output  (T=0)”  KPP  is  NOT  MET  since  the  model  is  not 
displayed  in  MS  Excel  using  the  format  that  the  TE  personnel  currently  use.  This 
issue  could  potentially  be  addressed  by  writing  code  in  MS  Excel  that  would 
convert  the  data  from  the  model  output  and  display  the  GAMS  output  in  a  format 
similar  to  the  one  that  the  test  planner’s  use. 

The  “Time  Period  Asset  Availability  (O)”  APA,  which  allows  for  test  assets 
to  be  assigned  as  unavailable  during  a  test  period,  is  NOT  VALIDATED.  The 
model  problem  formulation  and  the  model  input  files  show  that  the  model 
appears  to  support  this  APA,  but  the  model  has  not  been  tested. 

The  “Multiple  Asset  Types  (O)”  APA,  which  allows  low  priority  test  events 
to  go  beyond  the  test  period,  is  NOT  VALIDATED.  The  model  problem 
formulation  and  the  model  input  files  show  that  the  model  appears  to  support  this 
APA,  but  the  model  has  not  been  tested. 

The  “Model  in  MS  Excel  (O)”  APA,  which  requires  the  use  of  MS  Excel  as 
the  tool  to  perform  this  modelling,  is  NOT  MET.  The  model  uses  GAMS  with 
CMPLX  solver  as  the  modeling  tool. 

N.  MODEL  CRITICAL  OPERATIONAL  ISSUES 

This  research  explores  whether  optimization  can  aid  in  the  test  schedule 
development  process.  Five  model  schedules  have  been  evaluated  against  the 
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requirements  and  against  TE  schedules.  The  three  research  questions 
addressed  by  this  thesis,  considered  as  COIs,  are  now  assessed: 

a.  Can  a  test  scheduling  model  automate  the  test  schedule  development 
process?  As  illustrated  in  Table  29,  all  threshold  capabilities  are  met,  except  for 
one  KPP  relative  to  schedule  output.  The  model  generates  a  computer  printout, 
as  shown  in  Figure  10.  In  order  to  complete  the  test  schedule  development 
process,  the  model  output  must  be  manually  converted  into  detailed  schedules 
and  then  into  aggregated  schedules.  Without  achieving  this  KPP,  the  model 
cannot  be  considered  to  be  “automated”  without  additional  work.  However,  the 
additional  post-processing  automation  that  would  be  needed  is  achievable. 
Therefore,  we  can  state  that  a  test  scheduling  model  can  automate  the  test 
schedule  development  process,  and  we  have  assessed  this  COI  as  MET. 

b.  Can  a  test  scheduling  model  optimize  the  PMO  test  scheduling  activity 
to  provide  multiple  optimized  test  schedule  options?  The  model  uses  five 
schedule  sets  of  information  based  on  the  spreadsheet  that  the  TE  planners 
provided.  The  model  produces  a  printout,  as  shown  in  Figure  10.  The  model 
output  contains  detailed  information  for  five  schedules,  which  are  shown  for  this 
model  run  in  Appendix  B.  The  model  provides  detailed  daily  schedules  instead  of 
monthly  schedules  based  on  aggregated  data.  Aggregation  of  data  from  the  daily 
level  to  the  monthly  level  is  something  that  can  be  achieved.  As  stated  in  the  first 
COI,  the  post-processing  to  provide  the  data  in  the  TE  format  is  achievable. 
Therefore,  we  can  state  that  a  test  scheduling  model  can  optimize  test 
scheduling  by  providing  multiple  optimized  schedules,  and  we  have  assessed 
this  COI  as  MET. 

c.  Can  a  test  scheduling  model  determine  the  best  PMO  schedule  mix  of 

test  assets  and  test  facilities?  The  detailed  schedules  produced  track  to  the 

constraints  and  decision  rules  from  the  problem  statement  and  requirements. 

The  detailed  model  schedules  produced  by  the  model  and  contained  in  Appendix 

B  were  reasonable  detailed  schedules.  A  couple  of  the  longer  schedules 

generated  had  empty  cells  in  the  middle,  but  the  blocks  could  be  easily  shifted  to 
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the  left  with  additional  post-processing.  The  model  provides  detailed  daily 
schedules  instead  of  monthly  schedules  based  on  aggregated  data.  Aggregation 
of  data  from  the  daily  level  to  the  monthly  level  is  something  that  could  be 
accomplished.  As  stated  in  the  first  COI,  the  post-processing  to  provide  the  data 
in  the  TE  format  is  achievable.  The  evaluations  in  this  thesis  show  that  a  test 
scheduling  model  can  determine  the  mix  of  test  assets  and  test  facilities  and 
have  assessed  this  COI  as  MET. 

As  discussed  in  the  COIs,  while  the  model  itself  performed  well,  the  model 
output  required  manual  conversion  into  the  TE  planners  schedule  format.  The 
time  required  to  convert  the  data  from  the  current  model  form  into  a  useful 
schedule  is  such  that  the  model  is  not  operationally  effective,  nor  is  the  model 
operationally  suitable  in  its  current  form.  Additional  work  is  needed  in  the  area  of 
model  output  before  the  model  would  be  considered  to  be  operationally  effective 
and  operationally  suitable. 
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VI.  CONCLUSION 


Test  schedule  development  is  a  specialized  process  that  is  complex,  time- 
consuming,  and  iterative.  The  methodology  used  for  test  schedule  development 
depends  on  the  individual  test  planner  and  is  based  on  heuristics. 

This  research  establishes  the  heuristic  test  schedule  development 
process,  and  develops  COIs,  operational  needs,  and  detailed  requirements  for  a 
test  scheduling  model  using  constraint  and  rule-based  optimization.  The  resulting 
model  (Edwards  2015)  is  assessed  against  the  requirements,  operational  needs, 
and  COIs. 

Results  of  this  research  show  that  the  optimization  model  developed  by 
Shane  Edwards  (Edwards  2015)  meets  one  of  two  KPPs,  all  of  the  KSAs,  and 
the  threshold  APA  requirements.  The  model  achieves  the  following: 

•  provides  schedules  that  are  reasonably  close  to  what  the  TE  planners 
would  use 

•  minimizes  test  movement  between  test  venues  and  uses  a  test  venue 
distance  constraint 

•  allows  multiple  test  assets  to  be  tested  simultaneously 

•  places  test  events  based  on  priority 

•  allows  multiple  test  venues  to  be  used  at  the  same  time 

•  allows  test  venue  choice  by  test  event 

•  allows  multiple  test  events  to  occur  at  the  same  time 

•  has  test  event  predecessor  and  successor  relationship  constraints 

•  places  test  assets  based  on  a  deadline  for  each  priority  type 

•  minimizes  the  time  period  used  for  test  events 

•  allows  TE  personnel  to  enter  input  information  in  MS  Excel 
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Although  the  model  meets  the  majority  of  the  requirements,  the  results  of 
this  research  indicate  that  the  model  it  is  not  operationally  effective,  nor  is  it 
operationally  suitable  due  to  failure  to  meet  the  output  KPP.  However,  with 
additional  work  to  display  the  output  differently,  the  TE  test  schedule  optimization 
model  would  be  a  good  tool  for  the  TE  schedulers  to  use  to  improve  PMO  test 
schedule  development. 

A.  FURTHER  RESEARCH 

Further  research  could  mature  the  model  in  the  areas  of  additional 
functionality,  model  performance,  ease  of  use,  and  incorporation  into  PMO  tools, 
as  identified  by  the  model  developer  (Edwards  2015). 

Further  research  could  mature  the  model  to  add  RGT  to  the  model. 
Although  RGT  is  included  in  model  test  events,  RGT  is  a  completely  different 
type  of  testing  that  has  special  calculations  and  rules.  The  current  model  could 
be  enhanced  by  incorporating  these  RGT  calculation  and  rules  instead  of 
requiring  this  activity  to  be  performed  outside  of  the  model. 

Further  research  could  extend  the  model  to  address  the  entire  PMO 
schedule,  not  just  the  TE  portion.  A  PMO  schedule  model  would  benefit  PMOs 
since  PMOs  continuously  re-plan,  with  many  variables  in  play.  A  tool  to  help 
make  PMO  decisions  would  therefore  be  beneficial. 
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APPENDIX  A.  MODEL  INPUT  AND  OUTPUT 


The  input  for  the  model  contains  multiple  input  files  that  result  in  multiple 
schedule  outputs  in  a  single  file.  Included  in  this  appendix  are  data  input  files  and 
the  output  file  used  as  examples  to  verify  the  model.  All  of  these  files  are 
provided  by  Shane  A.  Edwards  (2015),  the  model  developer.  All  tables  in  this 
appendix  are  based  on  Shane  Edwards  model  input  files  (email  message  to 
author,  June  12,  2015). 

A.  INPUT 

The  model  input  consists  of  many  files  identified  as  follows: 

a.  Model  variable  p.  The  set  of  time  periods  that  the  test  assets  can  be 
assigned  to  test  venues  (v)  is  identified  in  this  file  (p.csv)  and  is  shown  in  Table 
30.  This  time  period  is  available  to  each  test  asset,  assuming  that  the  test  asset 
is  available  for  testing.  This  time  period  set  can  be  exceeded  for  low  priority  tests. 

Table  30.  Test  Period 

{  set  p  time  periods  } 
pOOl *p1 90 

b.  Model  variable  t.  The  tests  that  need  to  be  performed  by  the  test  assets 
are  located  in  this  file  (t.csv),  and  are  given  in  Tables  31 . 

Table  31.  Test  Events 

{  set  t  tasks  plus  explanatory  text} 

tOI  Tilt  Table 

t02  Side  Slopes 

t03  Standard  Obstacles 

t04  Land  Mode  Braking 

t05  Controlled  Maneuverability 

t06  Acceleration  Max.  and  Min.  Speeds 
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{  set  t  tasks  plus  explanatory  text} 
t07  Land  Mobility  Towing 
t08  Longitudinal  Slopes 

t09  Drawbar  Pull  and  Cooling  System  Heat  Balance 

tIO  Dead  Engine  Braking 

tl  1  Rolling  Resistance 

tl  2  Fuel  Consumptionjand 

tl  3  Initial  Inspection  and  Safety  Checkout  ATVB_CA 

tl 4  Fuel  Consumption_amphibious 

tl  5  Plow  in  testing 

tl  6  Controlled  Maneuverability 

tl  7  Speed  and  Powering 

tl  8  Maximum  Gross  Vehicle  Weight 

tl  9  Line  Passing  and  Towing 

t20  Surf  Transit 

t22  Ship  Operations 

t23  Multi_vehicle  operations 

t24  Navigation  and  Obstacle  Avoidance 

t25  Initial  Inspection  and  Safety  Checkout  ATC_MD 

t27  Land  Mode  lngress_Egress 

t28  Land  Mode  Ride  Quality 

t29  Land  Mode  Interior  Noise  and  Whole  Body 

Vibration 

t33  Emergency  Egress  land  and  water 

t36  Land  Mode  Automotive  Toxic  Fumes 

t37  Automotive  Toxic  Fumes  in  Water 

t39  Transportability 

t40  Physical  Characteristics 

t41  Climatic  Chambers 

t53  E3  testingjimited 

t56  RGT 

t57  RGT 

t58  RGT 

t59  RGT 

t60  RGT 

t61  RGT 

t62  RGT 

t63  Communications 
t64  Firing 


c.  Model  variable  a.  The  set  of  test  asset  types  that  are  being  tested  are 
located  in  this  file  (a.csv),  and  are  shown  in  Table  32.  In  this  particular  example, 

there  is  only  one  asset  type  identified  as  AAV. 
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Table  32.  Test  Asset  Types 


{  set  a  asset  types  } 
AAV 


d.  Model  variable  v.  The  test  facility  venues  are  located  in  this  file  (v.csv),  and 
are  shown  in  Table  33.  Although  test  venues  are  identified  in  this  file,  these  test 
facility  venues  may  or  may  not  be  used  because  use  depends  on  the  association  of 
the  test  venue  to  the  test  event  and  if  the  model  choses  the  test  venue. 


Table  33.  Test  Venues 

AT  C_M  D 

AVTB_CA 

EPG_AZ 

WSMFLNM 

YPG_AZ 

DPGJJT 

e.  Model  variable  i.  The  ordered  set  of  priorities  that  test  events  can  be  set 
to  is  located  in  this  file  (i.csv),  and  is  shown  in  Table  34. 


Table  34.  Test  Event  Priorities 

{  set  i  priorities  ordered  } 
high 
medium 
low 

f.  Model  variable  relationships  m_periods.  The  time  periods  needed  to 
move  between  the  test  venues  for  all  combinations  is  located  in  this  file 
(m_periods.csv),  and  is  shown  in  Table  35.  These  time  periods  are  inserted  into 
the  test  schedule  when  the  model  makes  the  decision  to  move  test  assets  to  test 
venues  (v).  The  first  column  is  the  test  venue  from  location,  the  second  column  is 
the  test  venue  to  location,  and  the  last  column  is  the  number  of  time  periods 
needed  to  move  to  the  test  venue. 
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Table  35.  Test  Venue  Movement  Time  Periods 


dummy 

dummy 

{  TABLE  m_periods(v  vp) } 
ATC_MD 

AVTB_CA 

ATC_MD 

EPG_AZ 

ATC_MD 

WSMRNM 

ATC_MD 

YPG_AZ 

ATC_MD 

DPG_UT 

AVTB_CA 

EPG_AZ 

AVTB_CA 

WSMRNM 

WSMRNM 

YPG_AZ 

WSMR_NM 

DPG_UT 

WSMR_NM 

AT  C_M  D 

YPG_AZ 

AVTB_CA 

YPG_AZ 

EPG_AZ 

YPG_AZ 

WSMRNM 

YPG_AZ 

DPGJJT 

YPG_AZ 

AT  C_M  D 

DPG_UT 

AVTB_CA 

DPG_UT 

EPG_AZ 

DPG_UT 

WSMRNM 

DPG_UT 

YPG_AZ 

DPG  UT 

ATC  MD 

m_periods 

{  upper  diagonal  only  assumed  symmetric  } 

6 

5 

5 

5 

5 

6 
3 
3 
3 
5 
3 
3 
3 
3 
5 
3 
3 
3 
3 
5 


g.  Model  variable  v_cap.  The  number  of  test  assets  that  each  test  facility 
venue  can  accommodate  is  located  in  this  file  (v_cap),  and  is  shown  in  Table  36. 
The  first  column  is  the  test  facility  venue  and  the  second  is  the  test  asset 
capacity. 


Table  36.  Test  Venue  Test  Asset  Capacity 


AT  C_M  D 

10 

AVTB_CA 

10 

EPG_AZ 

10 

WSMR_NM 

10 

YPG_AZ 

10 

DPG_UT 

10 
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h.  Model  variable  relationships  ti.  The  priorities  (i)  of  the  individual  test 
events  (t)  relationships  are  identified  in  this  file  (ti.csv),  and  are  shown  in  Table 
37.  The  first  column  is  the  test  event  and  the  second  is  the  associated  priority. 


Table  37.  Test  Event  Priority  Relationships 


toi 

high 

t02 

high 

t03 

high 

t04 

high 

t05 

high 

t06 

high 

t07 

medium 

t08 

high 

t09 

medium 

tio 

high 

til 

medium 

ti  2 

high 

ti  3 

high 

ti  4 

high 

ti  5 

high 

ti  6 

high 

ti  7 

high 

ti  8 

high 

ti  9 

high 

t20 

high 

t22 

high 

t23 

medium 

t24 

medium 

t25 

high 

t27 

high 

t28 

high 

t29 

high 

t33 

high 

t36 

high 

t37 

high 

t39 

medium 

t40 

medium 

t41 

medium 

t53 

medium 

t56 

high 

t57 

high 
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t58 

high 

t59 

high 

t60 

medium 

t61 

medium 

t62 

medium 

t63 

medium 

t64 

high 

i.  Model  variable  relationship  vt.  This  file  identifies  the  test  locations  that 
can  perform  each  test,  and  is  shown  in  Table  38.  The  first  column  is  the  test 
location  venue  (v).  The  second  column  is  the  test  event  (t)  identified  by  txx  where 
xx  is  the  number  of  the  test  as  identified  in  Table  31.  The  third  column  identifies 
the  test  asset  type  that  the  test  is  to  be  performed  on.  This  particular  file  has  only 
one  type. 

There  are  five  venues  identified  as  ATC_MD,  AVTB_CA,  WSMR_NM, 
YPG_AZ,  DPC_UT,  and  EPG_AZ.  In  this  input  file,  there  are  some  instances 
where  multiple  test  venues  are  identified  as  potential  test  venues  for  a  test  event. 


Table  38.  Test  Event  Test  Venue  Relationships 
{  SET  vt(vt)  facilities  capable  of  performing  each  test } 


AT  C_M  D 

toi 

{  vt  pairs  intentionally  omitted  as  test } 

1 

AT  C_M  D 

t02 

1 

AT  C_M  D 

t03 

1 

AT  C_M  D 

t04 

1 

AT  C_M  D 

t05 

1 

AT  C_M  D 

t06 

1 

AT  C_M  D 

t07 

1 

AT  C_M  D 

t08 

1 

AT  C_M  D 

t09 

1 

AT  C_M  D 

tio 

1 

AT  C_M  D 

til 

1 

AT  C_M  D 

tl  2 

1 

AVTB_CA 

tl  3 

1 

AVTB_CA 

tl  4 

1 

AVTB_CA 

tl  5 

1 
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{  SET  vt(vt)  facilities  capable  of  performing  each  test } 

{  vt  pairs  intentionally  omitted  as  test } 


AVTB_CA 

tl  6 

1 

AVTB_CA 

tl  7 

1 

AVTB_CA 

tl  8 

1 

AVTB_CA 

tl  9 

1 

AVTB_CA 

t20 

1 

AVTB_CA 

t22 

1 

AVTB_CA 

t23 

1 

AVTB_CA 

t24 

1 

AT  C_M  D 

t25 

1 

AT  C_M  D 

t27 

1 

AT  C_M  D 

t28 

1 

AT  C_M  D 

t29 

1 

AVTB_CA 

t33 

1 

AT  C_M  D 

t36 

1 

AVTB_CA 

t37 

1 

AT  C_M  D 

t39 

1 

AT  C_M  D 

t40 

1 

AT  C_M  D 

t41 

1 

YPG_AZ 

t41 

1 

WSMR_NM 

t53 

1 

AT  C_M  D 

t56 

1 

AVTB_CA 

t56 

1 

WSMR_NM 

t56 

1 

YPG_AZ 

t56 

1 

AT  C_M  D 

t57 

1 

AVTB_CA 

t57 

1 

WSMR_NM 

t57 

1 

YPG_AZ 

t57 

1 

AT  C_M  D 

t58 

1 

AVTB_CA 

t58 

1 

WSMR_NM 

t58 

1 

YPG_AZ 

t58 

1 

AT  C_M  D 

t59 

1 

AVTB_CA 

t59 

1 

WSMR_NM 

t59 

1 

YPG_AZ 

t59 

1 

AT  C_M  D 

t60 

1 

AVTB_CA 

t60 

1 

WSMR_NM 

t60 

1 

AVTB_CA 

t60 

1 

WSMR_NM 

t60 

1 

YPG  AZ 

t60 

1 
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{  SET  vt(vt)  facilities  capable  of  performing  each  test } 


AT  C_M  D 

t61 

{  vt  pairs  intentionally  omitted  as  test } 

1 

AVTB_CA 

t61 

1 

WSMR_NM 

t61 

1 

YPG_AZ 

t61 

1 

AT  C_M  D 

t62 

1 

AVTB_CA 

t62 

1 

WSMR_NM 

t62 

1 

YPG_AZ 

t62 

1 

AVTB_CA 

t63 

1 

AT  C_M  D 

t64 

1 

j.  Model  variable  relationship  rt.  The  set  of  test  event  precedence 
relationships  (t,  tp)  are  located  in  this  file  (rt.csv),  and  are  shown  in  Table  39.  The 
first  column  is  the  predecessor  test  event  and  the  second  column  is  the 
successor  test  event. 


Table  39.  Test  Event  Precedence  Relationships 

{  SET  rt(t  tp)  precedence  test  t  must  finish  before  test  tp  starts  } 


toi 

t02 

til 

t09 

tl  3 

tl  4 

tl  3 

tl  5 

tl  3 

tl  6 

tl  3 

tl  7 

tl  3 

tl  8 

tl  3 

tl  9 

tl  3 

t20 

tl  3 

t22 

tl  3 

t23 

tl  3 

t24 

tl  3 

t63 

tl  5 

tl  6 

tl  5 

tl  7 

tl  5 

tl  8 

t25 

toi 

t25 

t02 

t25 

f- 

o 

CO 
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{  SET  rt(t  tp)  precedence  test  t  must  finish  before  test  tp  starts  } 


t25 

t04 

t25 

t05 

t25 

t06 

t25 

t07 

t25 

t08 

t25 

t09 

t25 

tl  0 

t25 

til 

t25 

tl  2 

t25 

tl  4 

t25 

tl  5 

t25 

tl  6 

t25 

tl  7 

t25 

tl  8 

t25 

tl  9 

t25 

t20 

t25 

t22 

t25 

t23 

t25 

t24 

t25 

t27 

t25 

t28 

t25 

t29 

t25 

t33 

t25 

t36 

t25 

t37 

t25 

t39 

t25 

t40 

t25 

t41 

t25 

t53 

t25 

t64 

t40 

t39 

t53 

t22 

k.  Model  variable  relationship  t_data.  The  identification  of  how  many  test 
assets  are  needed  simultaneously  for  each  test  event  and  how  many  test  periods 
are  required  for  each  test  event  is  located  in  this  file  (t_data.csv),  and  is  shown  in 
Table  40.  The  information  is  given  for  four  different  input  amounts  for  the  number 
of  test  periods  needed  and  number  of  test  assets  needed  for  each  test  event 
(t_periods,  min,  mode,  and  max).  The  first  column  is  the  test  event.  The  second 
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column  is  the  number  of  test  assets  needed  to  execute  the  test  event.  The  third 
column  identifies  the  number  of  test  periods  needed  for  the  test  event  for  the 
t_periods  schedule.  The  fourth  column  identifies  the  number  of  test  periods 
needed  for  the  test  event  for  the  beta_min  schedule.  The  fifth  column  identifies 
the  number  of  test  periods  needed  for  the  test  event  for  the  beta_mode  schedule. 
The  sixth  column  identifies  the  number  of  test  periods  needed  for  the  test  event 
for  the  beta  max  schedule. 


Table  40.  Test  Event  Test  Period  Durations  and  Number  of  Test 

Assets  Needed  for  Test  Events 


{ t_data(t 

vehicles 

t_period 

min 

mode 

max} 

toi 

1 

1 

1 

2 

3 

t02 

1 

2 

3 

4 

7 

t03 

1 

4 

6 

8 

13 

t04 

1 

2 

3 

4 

7 

t05 

1 

2 

3 

4 

7 

t06 

1 

2 

3 

4 

7 

t07 

2 

2 

3 

4 

7 

t08 

1 

3 

4 

6 

10 

t09 

1 

10 

14 

20 

33 

tl  0 

1 

2 

3 

4 

7 

til 

1 

2 

3 

4 

7 

tl  2 

1 

4 

6 

8 

13 

tl  3 

3 

9 

13 

18 

30 

tl  4 

1 

5 

7 

10 

17 

tl  5 

1 

2 

3 

4 

7 

tl  6 

1 

2 

3 

4 

7 

tl  7 

1 

2 

3 

4 

7 

tl  8 

1 

2 

3 

4 

7 

tl  9 

2 

1 

1 

2 

3 

t20 

1 

1 

1 

2 

3 

t22 

1 

2 

3 

4 

7 

t23 

2 

3 

4 

6 

10 

t24 

1 

3 

4 

6 

10 

t25 

4 

9 

13 

18 

30 

t27 

1 

1 

1 

2 

3 

t28 

1 

5 

7 

10 

17 

t29 

1 

5 

7 

10 

17 

t33 

1 

2 

3 

4 

7 
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{ t_data(t  vehicles  t_period 

min 

mode 

max} 

t36 

1  5 

7 

10 

17 

t37 

1  5 

7 

10 

17 

t39 

1  20 

29 

40 

67 

t40 

1  5 

7 

10 

17 

t41 

1  20 

29 

40 

67 

t53 

1  15 

21 

30 

50 

t56 

1  22 

22 

22 

22 

t57 

1  22 

22 

22 

22 

t58 

1  22 

22 

22 

22 

t59 

1  22 

22 

22 

22 

t60 

1  22 

22 

22 

22 

t61 

1  22 

22 

22 

22 

t62 

1  22 

22 

22 

22 

t63 

1  21 

21 

32 

50 

t64 

1  15 

15 

22 

34 

I.  Model  variable  relationship  ta_data.  This  file  identifies  the  test  locations 
that  can  perform  each  test  (ta_data.csv),  and  is  shown  in  Table  41.  The  first 
column  is  the  test  event.  The  second  column  is  the  test  asset  type.  The  third 
column  identifies  whether  or  not  the  test  asset  is  subject  to  the  test,  where  “1” 
means  “yes.” 


Table  41 .  Test  Asset  Type  Test  Event  Relationships 


{ ta_data(t 

a 

test_subject 

a_type_req  }  { test  requirements  for  a  specific  asset  type  } 

toi 

AAV 

1 

1 

t02 

AAV 

1 

1  {  example  of  an  excluded  asset  type  from  a  test } 

t03 

AAV 

1 

1 

t04 

AAV 

1 

1 

t05 

AAV 

1 

1 

t06 

AAV 

1 

1 

t07 

AAV 

1 

1 

t08 

AAV 

1 

1 

t09 

AAV 

1 

1 

tl  0 

AAV 

1 

1 

til 

AAV 

1 

1 

tl  2 

AAV 

1 

1 

tl  3 

AAV 

1 

1 

tl  4 

AAV 

1 

1 
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{ ta_data(t 

a 

test_subject 

a_type_req  }  { test  requirements  for  a  specific  asset  type  } 

t15 

AAV 

1 

1 

tl  6 

AAV 

1 

1 

t17 

AAV 

1 

1 

tl  8 

AAV 

1 

1 

tl  9 

AAV 

1 

1 

t20 

AAV 

1 

1 

t22 

AAV 

1 

1 

t23 

AAV 

1 

1 

t24 

AAV 

1 

1 

t25 

AAV 

1 

1 

t27 

AAV 

1 

1 

t28 

AAV 

1 

1 

t29 

AAV 

1 

1 

t33 

AAV 

1 

1 

t36 

AAV 

1 

1 

t37 

AAV 

1 

1 

t39 

AAV 

1 

1 

t40 

AAV 

1 

1 

t41 

AAV 

1 

1 

t53 

AAV 

1 

1 

t56 

AAV 

1 

1 

t57 

AAV 

1 

1 

t58 

AAV 

1 

1 

t59 

AAV 

1 

1 

t60 

AAV 

1 

1 

t61 

AAV 

1 

1 

t62 

AAV 

1 

1 

t63 

AAV 

1 

1 

t64 

AAV 

1 

1 

m.  Model  variable  relationship  a_data.  The  identification  of  test  asset 
starting  locations,  and  test  period  unavailability  is  contained  in  this  file 
(a_data.csv),  and  is  shown  in  Table  42.  The  first  column  identifies  the  test  asset 
type.  The  second  column  identifies  the  starting  test  venue  for  the  test  asset  type. 
The  third  column  identifies  the  period  where  the  test  asset  at  the  test  venue 
starts.  The  fourth  column  identifies  how  many  test  assets  of  the  test  asset  type 
will  start  at  the  test  venue  at  the  test  period.  The  last  column  identifies  test  period 
unavailability. 


112 


Table  42.  Test  Asset  Test  Venue  Starting  Location  and  Test  Period 

Unavailability 

(a  v  p  a_rec  unavail } 

AAV  AT  C_M  D  pOOl  4  0 

AAV  AVTB_CA  pOOl  3  0 


n.  Model  variable  relationship  i_data.  The  identification  of  the  deadline  and 
the  associated  penalty  for  exceeding  the  deadline  is  given  in  this  file  (i_data.csv), 
and  is  shown  in  Table  43.  The  first  column  is  the  test  event  priority.  The  second 
column  is  the  deadline  test  period  that  the  test  event  needs  to  be  completed  by. 
The  last  column  is  the  penalty  used  by  the  model  if  the  model  goes  beyond  the 
deadline.  Low  priority  is  identified  as  not  having  a  deadline  (infinity  periods),  and 
not  having  a  penalty  (0).  This  addresses  the  requirement  that  for  low  priority  test 
asset  test  events  can  go  beyond  the  test  period.  This  is  accomplished  by  a 
variable  that  can  be  modified  to  have  a  hard  constraint,  which  provides  greater 
flexibility  in  the  model. 

Table  43.  Test  Event  Priority  Deadlines  and  Priority  Deadline  Penalties 


{i  deadline  penalty} 


high 

170 

200 

medium 

190 

100 

low 

inf 

0 

B.  OUTPUT 

Output  consists  of  restatement  of  some  of  the  input  data,  and  provides  the 
data  supporting  five  different  test  schedules  (beta_min,  beta_mode,  mean,  beta- 
Beta_max  and  t_periods).  The  size  of  this  file  is  too  large  to  include  in  this  thesis, 
but  is  available  upon  request.  The  output  file  used  in  this  thesis  is  based  on  a 
Shane  Edwards  model  output  file  (email  message  to  author,  June  12,  2015). 
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APPENDIX  B.  MODEL  OUTPUT  SCHEDULE  CONVERSIONS  AND 
REQUIREMENTS  ASSESSMENTS 


What  follows  in  this  appendix  are  the  five  manually  generated  schedules 
from  the  model  output  based  on  the  single  model  run.  These  schedules  have 
been  manually  converted  from  the  model  output  to  look  similar  to  the  TE 
schedules,  using  the  daily  time  periods  provided  by  the  model.  These  schedules 
are  used  to  develop  aggregated  monthly  schedules  in  the  main  body  of  this 
thesis  so  that  they  can  be  compared  against  the  TE  schedules.  The  Betajmax 
schedule  is  given  in  Figures  21-  28.  The  t_periods  schedule  is  given  in  Figures 
29-32.  The  beta_min  schedule  is  given  in  Figures  33-36.  The  beta_mode  is  given 
in  Figures  37-42.  The  beta_mean  is  given  in  Figures  43-48. 

In  these  tables,  test  assets  are  given  as  test  asset  TA1  through  TA7  on 
the  y-axis.  Test  periods  P001  through  Pxyz  are  given  on  the  x-axis  in  test  periods 
of  20  on  each  page  (e.g.  P001  -  P020,  P021  -  P040,  etc.).  The  cells  where  they 
intersect  in  the  interior  of  the  table  are  colored  based  on  test  venue  (AVTB  = 
blue,  ATC  =  green,  YPG  =  purple,  and  WSMR  =  salmon).  The  cells  where  they 
intersect  in  the  interior  of  the  table  are  given  a  font  based  on  priority  as  provided 
in  Table  37  of  Appendix  B  (high  =  bold,  medium  =  italic,  low  =  normal).  The  top 
line  of  the  cells  in  the  interior  of  the  table  are  the  functional  test  as  provided  in  the 
TE  schedule  data.  The  second  line  of  the  cells  in  the  interior  of  the  table  are  the 
actual  test  that  will  be  performed  on  that  test  asset  during  that  time  period  as 
provided  in  Table  31  of  Appendix  B  (t12  =  fuel  consumption,  t64  =  firing,  t59  = 
RGT,  t40  =  physical  characteristics,  t28  =  land  mode  ride  quality,  t53  =  E3 
testingjimited,  t58  =  RGT,  t15  =  plow  in  testing,  t37  =  Automotive  Toxic  Fumes 
in  Water). 

To  understand  this  better,  let’s  look  at  the  first  line  for  test  asset  TA1 .  TA1 
in  test  period  P001  is  performing  high  priority  initial  inspection  and  safety 
checkout  testing  at  ATC.  TA1  continues  this  testing  for  all  test  periods  through 
P020. 
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A. 


BETA  MAX  SCHEDULE 
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P016 
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Figure  21 .  First  Month  of  Model  Betajmax  Detailed  Schedule 
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Figure  22.  Second  Month  of  Model  Beta_max  Detailed  Schedule 
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Figure  23.  Third  Month  of  Model  Beta_max  Detailed  Schedule 
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Figure  24.  Fourth  Month  of  Model  Betajmax  Detailed  Schedule 
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Figure  25.  Fifth  Month  of  Model  Betajmax  Detailed  Schedule 
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Figure  26.  Sixth  Month  of  Model  Betajmax  Detailed  Schedule 
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Figure  27.  Seventh  Month  of  Model  Beta_max  Detailed  Schedule 
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Figure  28.  Eighth  Month  of  Model  Beta_max  Detailed  Schedule 
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Table  44  is  used  to  compare  the  input  file  to  the  schedule  developed  from  the 
model  output.  The  “Test  Event”  column  is  the  test  event  from  which  all  comparisons  are 
made.  “Test  Asset  Input”  is  the  number  of  test  assets  that  are  used  to  perform  the  test 
event  based  on  the  input  file.  “Actual  Test  Assets”  is  the  number  of  test  assets  used 
based  on  the  output  provided  and  what  is  shown  in  the  schedule.  “Test  Period  Input”  is 
the  number  of  test  periods  that  are  used  based  on  the  input  file.  “Actual  Test  Periods”  is 
the  number  of  test  periods  that  the  output  provided  and  what  is  shown  in  the  schedule. 
“Input  Location”  is  the  location  where  the  test  event  can  be  performed  based  on  the 
input  file.  “Actual  Location”  is  where  the  test  event  is  performed  based  on  the  output 
provided  and  what  is  shown  in  the  schedule.  The  “Successor”  column  provides  test 
events  that  are  successors  to  the  test  event  on  the  left  of  the  table.  The  “Successor 
Success?”  column  is  the  assessment  of  whether  that  test  event  is  started  after  the  test 
event  on  the  left  is  completed  or  not  with  an  answer  of  “yes”  or  “no.”  The  last  column, 
“Priority”  is  the  priority  from  the  input  file. 
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Table  44.  Model  Input  Files  to  Model  Betajnax  Detailed  Schedule  Comparison 


Test 

Test 

Actual 

Test 

Actual  Test 

Input 

Actual 

Successor 

Successor 

Priority 

Event 

Asset 

Test 

Period 

Periods 

Location 

Location 

Success? 

Input 

Assets 

Input 

toi 

1 

1 

3 

3 

ATC  MD 

ATC 

t02 

yes 

high 

t02 

1 

1 

7 

7 

ATC  MD 

ATC 

high 

t03 

1 

1 

13 

13 

ATC  MD 

ATC 

high 

t04 

1 

1 

7 

7 

ATC  MD 

ATC 

high 

t05 

1 

1 

7 

7 

ATC  MD 

ATC 

high 

t06 

1 

1 

7 

7 

ATC  MD 

ATC 

high 

t07 

2 

2 

7 

7 

ATC  MD 

ATC 

medium 

t08 

1 

1 

10 

10 

ATC  MD 

ATC 

high 

t09 

1 

1 

33 

33 

ATC  MD 

ATC 

medium 

tio 

1 

1 

7 

7 

ATC_MD 

ATC 

high 

til 

1 

1 

7 

7 

ATC  MD 

ATC 

t09 

yes 

medium 

tl  2 

1 

1 

13 

13 

ATC  MD 

ATC 

high 

tl  3 

3 

3 

30 

30 

AVTB_CA 

AVTB 

tl 4,  tl 5,  tl 6,  tl 7,  tl  8,  tl  9, 
t20,  t22,  t23,  t24,  t63 

yes 

high 

tl  4 

1 

1 

17 

17 

AVTB  CA 

AVTB 

high 

tl  5 

1 

1 

7 

7 

AVTB_CA 

AVTB 

tl  6,  tl  7,  tl  8 

yes 

high 

tl  6 

1 

1 

7 

7 

AVTB  CA 

AVTB 

high 

tl  7 

1 

1 

7 

7 

AVTB  CA 

AVTB 

high 

tl  8 

1 

1 

7 

7 

AVTB  CA 

AVTB 

high 

tl  9 

2 

1 

3 

3 

AVTB  CA 

AVTB 

high 

t20 

1 

1 

3 

3 

AVTB  CA 

AVTB 

high 

t22 

1 

1 

7 

7 

AVTB  CA 

AVTB 

high 

t23 

2 

2 

10 

10 

AVTB  CA 

AVTB 

medium 

t24 

1 

1 

10 

10 

AVTB_CA 

AVTB 

medium 

t25 

4 

4 

30 

30 

ATC_MD 

ATC 

tOI ,  t02,  t03,  t04,  t05,  t06, 

yes 

high 

t07,  t08,t09,  tio,  til,  t12, 
tl  4,  tl  5,  tl  6,  tl  7,  tl  8,  tl  9, 
t20,  t22,  t23,  t24,  t27,  t28, 
t29,  t33,  t36,  t39,  t40,  t41 , 
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Test 

Test 

Actual 

Test 

Actual  Test 

Input 

Actual  Successor 

Successor  Priority 

Event 

Asset 

Input 

Test 

Assets 

Period 

Input 

Periods 

Location 

Location 

Success? 

t53,  t64 


t27 

1 

1 

3 

3 

ATC  MD 

ATC 

high 

t28 

1 

1 

17 

17 

ATC  MD 

ATC 

high 

t29 

1 

1 

17 

17 

ATC  MD 

ATC 

high 

t33 

1 

1 

7 

7 

AVTB  CA 

AVTB 

high 

t36 

1 

1 

17 

17 

ATC  MD 

ATC 

high 

t37 

1 

1 

17 

17 

AVTB  CA 

AVTB 

high 

t39 

1 

1 

67 

67 

ATC  MD 

ATC 

medium 

t40 

1 

1 

17 

17 

ATC  MD 

ATC 

t39 

yes 

medium 

t41 

1 

1 

67 

67 

ATC  MD 

YPG  AZ 

WSMR 

medium 

t53 

1 

1 

50 

50 

WSMR  NM 

WSMR 

t22 

yes 

medium 

t56 

1 

1 

22 

22 

ATC  MD 

ATC 

high 

AVTBCA 
WSMRNM 
YPG  AZ 


t57 

1 

1 

22 

22 

ATC  MD 
AVTB  CA 
WSMR  NM 
YPG  AZ 

ATC 

high 

t58 

1 

1 

22 

22 

ATC  MD 
AVTB  CA 
WSMR  NM 
YPG  AZ 

AVTB 

high 

t59 

1 

22 

22 

22 

ATC  MD 
AVTB  CA 
WSMR  NM 
YPG  AZ 

ATC 

high 

t60 

1 

1 

22 

22 

ATC  MD 
AVTB  CA 
WSMR  NM 
YPG  AZ 

ATC 

medium 
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Test 

Event 

Test 

Asset 

Input 

Actual 

Test 

Assets 

Test 

Period 

Input 

Actual  Test 
Periods 

Input 

Location 

Actual  Successor 

Location 

Successor 

Success? 

Priority 

t61 

1 

1 

22 

22 

ATC  MD 
AVTB  CA 
WSMR  NM 
YPG  AZ 

ATC 

medium 

t62 

1 

1 

22 

22 

ATC  MD 
AVTB  CA 
WSMR  NM 
YPG  AZ 

ATC 

medium 

t63 

1 

1 

50 

50 

AVTB  CA 

AVTB 

medium 

t64 

1 

34 

34 

ATC  MD 

ATC 

high 
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Using  the  information  provided  in  the  Beta_max  schedule  and  the 
comparison  table,  the  verification  assessment  results  for  Beta_max  schedule 
generated  by  the  model  is  given  in  the  Table  45  in  the  “Verification  &  Rationale” 
column,  which  is  relative  to  Table  4  requirements  that  are  repeated  here. 


Table  45.  Model  Controls  and  Constraints  Betajmax  Schedule 

Verification 


Para  # 

Para  Title 

Requirement 

Verification  Criteria 

Verification  & 
Rationale 

2.0 

Controls  and 
Constraints 

Not  Applicable 

Not  Applicable 

Not  Applicable 

2.1 

Test  Period  (T=0) 

The  TE  RSCSP  model 
schedule  shall  be 
constrained  by  the  test 
period. 

The  requirement 
shall  be  verified  by 
confirming  that  the 
schedule  is  within 
the  test  period  given 
in  the  test  period 
input  file. 

MET 

Test  period  is 

180  and  model 
test  period  is 

159  days. 

2.1.1 

One  Test  Event  on 
Test  Asset  during 

Time  Period  (T=0) 

The  TE  RSCSP  model 
shall  not  allow  more 
than  one  test  event 
during  a  time  period 
on  a  test  asset. 

The  requirement 
shall  be  verified  by 
confirming  that  the 
schedule  does  not 
show  more  than  one 
test  event  in  a  time 
period  on  a  test 
asset. 

PARTIALLY 

MET 

Output  file  does 
not  assign  a 
test  asset,  but 
in  aggregate 
does  not 
exceed  the 
number  of  test 
assets. 

2.1.2 

Place  all  Test  Events 
in  Test  Period  (T=0) 

The  TE  RSCSP  model 
shall  place  all 
identified  test  events  in 
a  test  period. 

The  requirement 
shall  be  verified  by 
confirming  that 
schedule  places  all 
identified  test 
events  in  a  test 
period. 

MET 

Actual  test 
events  from 
input  file  are  all 
placed  in  test 
periods. 

2.1.3 

Test  Event  Test 

Period  Durations 
(T=0) 

The  TE  RSCSP  model 
shall  use  the  test 
event  test  period 
durations  input. 

The  requirement 
shall  be  verified  by 
confirming  that 
schedule  test  event 
test  period  durations 
matches  the  test 
events  test  period 
input  file. 

MET 

Actual  test 
event  test 
periods  tracks 
to  input  tile. 
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Para  # 

Para  Title 

Requirement 

Verification  Criteria 

Verification  & 
Rationale 

2.2 

Test  Events  (T=0) 

The  TE  RSCSP  model 
schedule  shall  be 
constrained  by  the  test 
events. 

The  requirement 
shall  be  verified  by 
confirming  that  the 
schedule  includes 
all  test  events 
included  in  the  test 
events  input  file. 

MET 

2.3 

Test  Asset  Type  (0) 

The  TE  RSCSP  model 
shall  use  test  asset 
type  inputs. 

The  requirement 
shall  be  verified  by 
confirming  that 
schedule  results 
places  test  events 
on  the  applicable 
test  asset  type 
based  on  the  input 
file. 

NOT  VERIFIED 
Input  files  did 
not  use  more 
than  one  test 
asset  type. 

2.4 

Test  Venues  (T=0) 

The  TE  RSCSP  model 
shall  use  test  venues 
input. 

The  requirement 
shall  be  verified  by 
confirming  that  the 
schedule  includes 
only  the  test  venues 
included  in  the  test 
venues  input  file. 

MET 

Actual  test 
venues  used 
track  to  input 
file. 

2.5 

Test  Event  Priorities 

Not  Applicable 

Not  Applicable 

Not  Applicable 

2.5.1 

Test  Event  Priority 
Relationships  (T=0) 

The  TE  RSCSP  model 
shall  be  constrained 
by  the  test  event 
priority  relationships 

The  requirement 
shall  be  verified  by 
confirming  that 
schedule  results  are 
constrained  by  the 
test  event  priority 
relationships  input 
file. 

PARTIALLY 

MET 

In  most  cases, 
MET.  There  is 
an  issue  with 
t33  and  t37 
medium 
happening 
before  high 
priority  tests. 

2.5.2 

High  to  Medium 
Priority  Relationship 
(T=0) 

The  TE  RSCSP  model 
high  priority  test 
events  shall  be  started 
before  medium  priority 
test  events. 

The  requirement 
shall  be  verified  by 
confirming  that 
schedule  starts  high 
priority  test  events 
before  medium 
priority  test  events. 

PARTIALLY 

MET 

2.5.3 

Medium  to  Low 

Priority  Relationship 
(T=0) 

The  TE  RSCSP  model 
medium  priority  test 
events  shall  be  started 
before  low  priority  test 
events. 

The  requirement 
shall  be  verified  by 
confirming  that 
schedule  starts 
medium  priority  test 
events  before  low 
priority  test  events. 

NOT  VERIFIED 
Model  run  did 
not  include  any 
low  priority  test 
events. 
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Para  # 

Para  Title 

Requirement 

Verification  Criteria 

Verification  & 
Rationale 

2.5.4 

High  to  Low  Priority 
Relationships  (T=0) 

The  TE  RSCSP  model 
high  priority  test 
events  shall  be  started 
before  low  priority  test 
events. 

The  requirement 
shall  be  verified  by 
confirming  that 
schedule  starts  high 
priority  test  events 
before  low  priority 
test  events. 

NOT  VERIFIED 
Model  run  did 
not  include  any 
low  priority  test 
events. 

2.6 

Test  Venue 

Movement  Test 
Periods  (T=0) 

The  TE  RSCSP  model 
shall  be  constrained 
by  the  test  venue 
movement  test 
periods. 

The  requirement 
shall  be  verified  by 
confirming  that 
schedule  uses  the 
test  venue 
movement  test 
periods  based  on 
the  input  file. 

MET 

Data  shows 
that  when 
moving  venues, 
the  input  data 
matches  the 
input  file. 

2.6.1 

Add  Time  Period 
from  Test  Venue 
Movement  (T=0) 

The  TE  RSCSP  model 
shall  add  time  periods 
to  the  schedule  based 
on  the  distance 
between  test  venues. 

The  requirement 
shall  be  verified  by 
confirming  that 
schedule  results 
show  that  the  time 
periods  added  to 
the  schedule  when 
the  test  asset 
moves  to  a  new  test 
venue  are  based  on 
the  input  file. 

MET 

2.6.2 

Schedule  Test  Event 
on  Available  Test 
Venues  (T=0) 

The  TE  RSCSP  model 
shall  not  schedule  a 
test  event  at  a  test 
venue  that  does  not 
perform  that  test 
event. 

The  requirement 
shall  be  verified  by 
confirming  that 
schedule  results 
show  that  the  test 
events  are 
scheduled  only  on 
available  test 
venues  based  on 
the  input  file. 

MET 

Test  venues 
used  track  to 
test  venues 
allowed  based 
on  the  input  file. 

2.7 

Test  Venue  Test 

Asset  Capacity 
(T=0) 

The  TE  RSCSP  model 
shall  be  constrained 
by  test  venue  test 
asset  capacity  input. 

The  requirement 
shall  be  verified  by 
confirming  that  the 
number  of  test 
assets  located  at 
each  test  venue  on 
the  schedule  does 
not  exceed  the 
capacity  identified  in 
the  input  file. 

NOT  VERIFIED 
The  number  of 
test  assets 
assigned  to  a 
test  venue  by 
the  model  did 
not  exceed  the 
amount  in  the 
file,  but  the 
numbers  in  the 
file  (10)  did  not 
really  check  the 
model. 
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Para  # 

Para  Title 

Requirement 

Verification  Criteria 

Verification  & 
Rationale 

2.8 

Test  Asset  Test 

Venue  Starting 
Location  (T=0) 

The  TE  RSCSP  model 
shall  be  constrained 
by  the  test  asset  test 
venue  starting  location 
input. 

The  requirement 
shall  be  verified  by 
confirming  that 
schedule  test  assets 
start  at  the  venues 
identified  in  the  test 
asset  test  venue 
input  file. 

MET 

2.9 

Test  Event  Test 

Venue  Relationships 
(T=0) 

The  TE  RSCSP  model 
shall  be  constrained 
by  the  test  event  test 
venue  relationships 
input. 

The  requirement 
shall  be  verified  by 
confirming  that 
schedule  test 
events  are 
performed  at  the 
test  venues 
identified  in  the  test 
event  test  venue 
relationships  input 
file. 

MET 

2.10 

Test  Event 
Precedence 
Relationships  (T=0) 

The  TE  RSCSP  model 
shall  be  constrained 
by  test  event 
precedence 
relationships  input. 

The  requirement 
shall  be  verified  by 
confirming  that 
schedule  results  are 
constrained  by  the 
test  event 
precedence 
relationships  input 
file. 

MET 

Data  shows 
that 

predecessor 
test  events  in 
the  schedule 
occur  before 
successor  test 
events. 

2.10.1 

Test  Event 
Predecessor  First 
(T=0) 

The  TE  RSCSP  model 
shall  perform 
predecessor  test 
events  prior  to  test 
event  successor  test 
events. 

The  requirement 
shall  be  verified  by 
confirming  that 
predecessor  test 
events  are  on  the 
schedule  before 
successor  test 
events  regardless  of 
test  event  priority. 

MET 

Data  shows 
that 

predecessor 
test  events  in 
the  schedule 
occur  before 
successor  test 
events. 

2.10.2 

Test  Event 

Successor  Second 
(T=0) 

The  TE  RSCSP  model 
shall  not  perform 
successor  test  events 
prior  to  test  event 
predecessor  test 
events. 

The  requirement 
shall  be  verified  by 
confirming  that 
successor  test 
events  are  on  the 
schedule  before 
predecessor  test 
events  regardless  of 
test  event  priority. 

MET 
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Para  # 

Para  Title 

Requirement 

Verification  Criteria 

Verification  & 
Rationale 

2.11 

Number  of  Test 
Assets  Needed  for 
Test  Events  (T=0) 

The  TE  RSCSP  model 
shall  use  the  number 
of  test  assets  needed 
for  test  events  input. 

The  requirement 
shall  be  verified  by 
confirming  that  the 
schedule  uses  the 
number  of  test 
assets  needed  for 
test  events  input 
file. 

MET 

2.12 

Low  Priority 

Schedule 
Relationship  (0) 

The  TE  RSCSP  model 
low  priority  test  events 
shall  be  allowed  to  go 
beyond  the  test  period. 

The  requirement 
shall  be  verified  by 
confirming  that 
schedule  results 
match  the  input  file. 

NOT  VERIFIED 
Low  priority  test 
events  are  not 
used  in  this 
model  run. 

2.13 

Test  Asset 
Unavailability  (0) 

The  TE  RSCSP  model 
shall  be  constrained 
by  the  test  asset 
unavailability. 

The  requirement 
shall  be  verified  by 
confirming  that 
schedule  places  test 
assets  only  on 
available  test  assets 
based  on  the  input 
file. 

NOT  VERIFIED 
Data  for  Test 
Asset 

Unavailability  is 
not  included  in 
the  model  run. 

2.14 

Test  Venue 
Unavailability  (0) 

The  TE  RSCSP  model 
shall  be  constrained 
by  test  venue 
unavailability. 

The  requirement 
shall  be  verified  by 
confirming  that 
schedule  results  are 
constrained  by  the 
input  file. 

NOT  VERIFIED 
Data  for  Test 
Venue 

Unavailability  is 
not  included  in 
the  model  run. 

2.15 

Test  Event  Priority 
Deadlines  (T=0) 

The  TE  RSCSP  model 
shall  be  constrained 
by  the  high  and 
medium  test  event 
priority  deadlines. 

The  requirement 
shall  be  verified  by 
confirming  that 
schedule  results  are 
constrained  by  the 
input  file. 

PARTIALLY 

MET 

2.15.1 

High  Priority  to 
Deadline 

Relationship  (T=0) 

TheTE  RSCSP  model 
shall  complete  high 
priority  test  events 
before  high  priority 
deadlines. 

The  requirement 
shall  be  verified  by 
confirming  that 
schedule  results 
start  high  priority 
test  events  before 
the  high  priority 
deadline. 

MET 

2.15.2 

Medium  Priority  to 
Deadline 

Relationship  (T=0) 

The  TE  RSCSP  model 
shall  complete  medium 
priority  test  events 
before  medium  priority 
deadlines. 

The  requirement 
shall  be  verified  by 
confirming  that 
schedule  results 
start  medium 
priority  test  events 
before  the  medium 
priority  deadline. 

NOT  VERIFIED 
This  model  run 
used  1 90  for  the 
total  test  periods 
and  the  medium 
test  periods, 
which  means 
that  it  is  not 
verified. 
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Para  # 

Para  Title 

Requirement 

Verification  Criteria 

Verification  & 
Rationale 

2.15.3 

High  Priority  to  Test 
Period  Relationship 
(T=0) 

The  TE  RSCSP  model 
shall  complete  high 
priority  test  events 
before  the  test  period 
completes. 

The  requirement 
shall  be  verified  by 
confirming  that 
schedule  results 
complete  high 
priority  test  events 
before  the  test 
period  completes. 

MET 

High  priority  test 
events  end  at  95. 
Schedule  ends 
at  test  period 

159.  Test  period 
ends  at  190. 

2.15.4 

Medium  Priority  to 
Test  Period 
Relationship  (T=0) 

The  TE  RSCSP  model 
shall  complete  medium 
priority  test  events 
before  the  test  period 
completes. 

The  requirement 
shall  be  verified  by 
confirming  that 
schedule  results 
complete  medium 
priority  test  events 
before  the  test 
period  completes. 

MET 

Medium  priority 
test  events  end 
at  test  period 

159.  Schedule 
ends  at  test 
period  159. 

Test  period  ends 
at  190. 

The  results  of  the  model  controls  and  constraints  requirements  verification 
assessment  for  Betajmax  indicate  that  the  majority  of  the  threshold  model 
controls  and  constraints  requirements  in  Table  4  are  MET,  except  for  some  that 
are  PARTIALLY  MET  and  some  that  are  NOT  VERIFIED.  There  are  no  controls 
and  constraints  requirements  that  are  verified  to  be  NOT  MET. 

Partially  Met  requirements.  Data  used  for  verification  for  time  period, 
priority  deadlines  did  not  fully  test  the  time  period  due  to  the  values  used.  The 
following  are  considered  to  be  PARTIALLY  MET,  and  require  an  additional 
verification  event  in  order  to  be  fully  assessed:  One  Test  Event  on  Test  Asset 
during  Time  Period,  Test  Event  Priority  Relationships,  High  to  Medium  Priority 
Relationship,  and  Test  Event  Priority  Deadlines. 

Not  Verified  requirements.  Data  did  not  include  low  priority  test  events, 
test  asset  type,  test  venue  capacity,  test  asset  unavailability,  test  venue 
unavailability,  and  medium  deadline.  The  following  are  considered  to  be  NOT 
VERIFIED,  and  require  an  additional  verification  event  in  order  to  be  assessed: 
Medium  to  Low  Priority  Relationship,  High  to  Low  Priority  Relationships,  Test 
Venue  Test  Asset  Capacity,  Test  Asset  Type  Test  Event  Relationships,  Test 
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Asset  Unavailability,  Test  Venue  Unavailability,  Medium  Priority  to  Deadline 
Relationship,  and  Low  Priority  Schedule  Relationship. 
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B.  T  PERIODS  SCHEDULE 


Time  Period 

Test 

Asset 

P001 

P002 

P003 

P004 

P005 

P006 

P007 

P008 

P009 

P010 

P011 

P012 

P013 

P014 

P015 

P016 

P017 

P018 

P019 

P020 

TA1 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

ATCto 

ATCto 

ATCto 

ATCto 

ATCto 

S 

S 

S 

S 

S 

S 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

WSMR 

WSMR 

WSMR 

WSMR 

WSMR 

f  53 

t53 

t53 

t53 

t53 

t53 

TA2 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t06 

t06 

toi 

tl2 

tl2 

tl2 

tl2 

t02 

t02 

til 

til 

TA3 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

LM 

LM 

RGT1 

RGT1 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t27 

t40 

t40 

t40 

t40 

t40 

t04 

t04 

t57 

t57 

TA4 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

F 

F 

F 

F 

F 

F 

F 

F 

F 

F 

F 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t64 

t64 

t64 

t64 

t64 

t64 

t64 

t64 

t64 

t64 

t64 

TA5 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

WM 

WM 

WM 

WM 

WM 

tl3 

tl3 

tl3 

tl3 

tl3 

tl3 

tl3 

tl3 

tl3 

tl9 

tl4 

tl4 

tl4 

tl4 

TA6 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

WM 

S/HF 

S/HF 

S/HF 

S/HF 

tl3 

tl3 

tl3 

tl3 

tl3 

tl3 

tl3 

tl3 

tl3 

tl9 

t37 

t37 

t37 

t37 

TA7 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

WM 

RGT1 

RGT1 

RGT1 

RGT1 

tl3 

tl3 

tl3 

tl3 

tl3 

tl3 

tl3 

tl3 

tl3 

t20 

t56 

t56 

t56 

t56 

Figure  29.  First  Month  of  Model  t_periods  Detailed  Schedule 
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Time  Period 

Test 

Asset 

P021 

P022 

P023 

P024 

P025 

P026 

P027 

P028 

P029 

P030 

P031 

P032 

P033 

P034 

P035 

P036 

P037 

P038 

P039 

P040 

5 

5 

5 

S 

S 

S 

S 

S 

S 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

t53 

t53 

t53 

tS3 

tS3 

t53 

tS3 

tS3 

t53 

t60 

t60 

t60 

t60 

t60 

t60 

t60 

t60 

t60 

t60 

t60 

TA2 

LM 

LM 

LM 

LM 

LM 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

LM 

LM 

LM 

LM 

LM 

LM 

t08 

t08 

t08 

tio 

tio 

t28 

t28 

t28 

t28 

t28 

t07 

t07 

t09 

t09 

t09 

t09 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

1  Aj 

t57 

t57 

t57 

t57 

t57 

t57 

t57 

t57 

t57 

t57 

t57 

t57 

t57 

t57 

t57 

t57 

t57 

t57 

t57 

t57 

TA4 

F 

F 

F 

F 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

LM 

LM 

LM 

LM 

t64 

t64 

t64 

t64 

t36 

t36 

t36 

t36 

t36 

t05 

t05 

t07 

t07 

WM 

WM 

WM 

WM 

WM 

WM 

WM 

WM 

WM 

WM 

WM 

WM 

WM 

WM 

1  Aj 

tl4 

t24 

t24 

t24 

tl5 

tl5 

tl8 

tl8 

tl7 

tl7 

tl6 

tl6 

t22 

t22 

S/HF 

WM 

WM 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

1  Ab 

t37 

t33 

t33 

t58 

t58 

t58 

t58 

t58 

t58 

t58 

t58 

t58 

t58 

t58 

t58 

t58 

t58 

t58 

t58 

t58 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT2 

RGT2 

t56 

t56 

t56 

t56 

t56 

t56 

t56 

t56 

t56 

t56 

t56 

t56 

t56 

t56 

t56 

t56 

t56 

t56 

t61 

t61 

Figure  30.  Second  Month  of  Model  t_periods  Detailed  Schedule 
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Time  Period 

Test 

Asset 

P041 

P042 

P043 

P044 

P045 

P046 

P047 

P048 

P049 

P050 

P051 

P052 

P053 

P054 

P055 

P056 

P057 

P058 

P059 

P060 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

t60 

t60 

t60 

t60 

t60 

t60 

t60 

t60 

t60 

t60 

t60 

LM 

LM 

LM 

LM 

LM 

LM 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

t09 

t09 

t09 

t09 

t09 

t09 

t41 

t41 

t41 

t41 

t41 

t41 

t41 

t41 

t41 

t41 

t41 

t41 

t41 

t41 

LM 

LM 

LM 

LM 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

1  Aj 

t03 

t03 

t03 

t03 

t59 

t59 

t59 

t59 

t59 

t59 

t59 

t59 

t59 

t59 

t59 

t59 

t59 

t59 

t59 

t59 

TA4 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

t29 

t29 

t29 

t29 

t29 

t39 

t39 

t39 

t39 

t39 

t39 

t39 

t39 

t39 

t39 

t39 

t39 

t39 

t39 

t39 

RGT2 

RGT2 

RGT2 

RGT2 

RGT2 

RGT2 

RGT2 

RGT2 

RGT2 

RGT2 

RGT2 

RGT2 

RGT2 

RGT2 

RGT2 

RGT2 

RGT2 

RGT2 

RGT2 

RGT2 

1  Aj 

t62 

t62 

t62 

t62 

t62 

t62 

t62 

t62 

t62 

t62 

t62 

t62 

t62 

t62 

t62 

t62 

t62 

t62 

L62 

t62 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

C 

C 

C 

C 

C 

C 

C 

C 

C 

C 

C 

C 

C 

C 

C 

t58 

t58 

t58 

t58 

t58 

t63 

t63 

t63 

t63 

t63 

t63 

t63 

t63 

t63 

t63 

t63 

t63 

t63 

t63 

t63 

RGT2 

RGT2 

RGT2 

RGT2 

RGT2 

RGT2 

RGT2 

RGT2 

RGT2 

RGT2 

RGT2 

RGT2 

RGT2 

RGT2 

RGT2 

RGT2 

RGT2 

RGT2 

RGT2 

RGT2 

1  A/ 

t61 

t61 

t61 

t61 

t61 

t61 

t61 

t61 

t61 

t61 

t61 

t61 

t61 

t61 

t61 

t61 

t61 

t61 

t61 

t61 

Figure  31 .  Third  Month  of  Model  t_periods  Detailed  Schedule 
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Time  Period 

Test 

P061 

P062 

P063 

P064 

P065 

P066 

P06 

P06 

P06 

P07 

P07 

P07 

P07 

P07 

P07 

P07 

P07 

P07 

P07 

P08 

Asse 

t 

7 

8 

9 

0 

1 

2 

3 

4 

5 

6 

7 

8 

9 

0 

TA1 

TA2 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

t41 

t41 

t41 

t41 

t41 

t41 

RGT 

RGT 

RGT 

RGT 

RGT 

RGT 

TA3 

1 

1 

1 

1 

1 

1 

t59 

t59 

t59 

t59 

t59 

t59 

TA4 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

t39 

t39 

t39 

t39 

t39 

TA5 

RGT 

RGT 

WM 

WM 

WM 

2 

t62 

2 

t62 

t23 

t23 

t23 

TA6 

C 

C 

C 

C 

C 

C 

t63 

t63 

t63 

t63 

t63 

t63 

TA7 

WM 

WM 

WM 

t23 

t23 

t23 

Figure  32.  Fourth  Month  of  Model  t_periods  Detailed  Schedule 
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Table  46.  Model  Input  Files  to  Model  t_periods  Detailed  Schedule  Comparison 


Test 

Event 

Test 

Asset 

Input 

ActUc 

Test 

Asset 

tl  Test 

Period 
s  Input 

Actual 

Test 

Periods 

Input 

Location 

Actual 

Location 

Successor 

Successor 

Success? 

Priority 

Priority 

Notes 

toi 

1 

1 

1 

1 

ATC_MD 

ATC 

t02 

Yes 

high 

t02 

1 

1 

2 

2 

ATC_MD 

ATC 

high 

t03 

1 

1 

4 

4 

ATC_MD 

ATC 

high 

t04 

1 

1 

2 

2 

ATC_MD 

ATC 

high 

t05 

1 

1 

2 

2 

ATC_MD 

ATC 

high 

t06 

1 

1 

2 

2 

ATC_MD 

ATC 

high 

t07 

2 

2 

2 

2 

ATC_MD 

ATC 

medium 

performed 
before 
high  tasks 

t08 

1 

1 

3 

3 

ATC_MD 

ATC 

high 

t09 

1 

1 

10 

10 

ATC_MD 

ATC 

medium 

performed 
before 
high  tasks 

tio 

1 

1 

2 

2 

ATC_MD 

ATC 

high 

til 

1 

1 

2 

2 

ATC_MD 

ATC 

t09 

Yes 

medium 

performed 
before 
high  tasks 

t12 

1 

1 

4 

4 

ATC_MD 

ATC 

high 

t13 

3 

3 

9 

9 

AVTBCA 

AVTB 

tl  4,  tl  5,  tl  6,  tl  7, 
tl  8,  tl  9,  t20,  t22, 
t23,  t24,  t63 

Yes 

high 

t14 

1 

1 

5 

5 

AVTB_CA 

AVTB 

high 
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Test 

Event 

Test 

Asset 

Input 

Actus 

Test 

Asset 

il  Test 

Period 
s  Input 

Actual 

Test 

Periods 

Input 

Location 

t15 

1 

1 

2 

2 

AVTB_CA 

t16 

1 

1 

2 

2 

AVTB_CA 

t17 

1 

1 

2 

2 

AVTBCA 

t18 

1 

1 

2 

2 

AVTB_CA 

t19 

2 

2 

1 

1 

AVTBCA 

t20 

1 

1 

1 

1 

AVTB_CA 

t22 

1 

1 

2 

2 

AVTB_CA 

t23 

2 

2 

3 

3 

AVTB_CA 

t24 

1 

1 

3 

3 

AVTB_CA 

t25 

4 

4 

9 

9 

ATC_MD 

t27 

1 

1 

1 

1 

ATC_MD 

t28 

1 

1 

5 

5 

ATC_MD 

t29 

1 

1 

5 

5 

ATC_MD 

t33 

1 

1 

2 

2 

AVTB_CA 

Actual 

Location 

Successor 

Successor 

Success? 

Priority 

Priority 

Notes 

AVTB 

tl  6,  tl  7,  tl  8 

Yes 

high 

AVTB 

high 

AVTB 

high 

AVTB 

high 

AVTB 

high 

AVTB 

high 

AVTB 

high 

performed 
before 
high  tasks 

AVTB 

medium 

AVTB 

medium 

performed 
before 
high  tasks 

ATC 

tOI,  t02,  t03,  t04, 
t05,  t06,  t07,  t08, 
t09,  tl 0,  til,  tl 2, 
tl  4,  tl  5,  tl  6,  tl  7, 
tl  8,  tl  9,  t20,  t22, 
t23,  t24,  t27,  t28, 
t29,  t33,  t36,  t39 
t40,  t41 ,  t53,  t64 

Yes.  Note 
that  this 
should  only 
be  for  ATC, 
not  AVTB 
tests 

high 

ATC 

high 

ATC 

high 

ATC 

high 

AVTB 

high 
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Test 

Event 

Test 

Asset 

Input 

Actus 

Test 

Asset 

il  Test 

Period 
s  Input 

Actual 

Test 

Periods 

Input 

Location 

t36 

1 

1 

5 

5 

ATC_MD 

t37 

1 

1 

5 

5 

AVTB_CA 

t39 

1 

1 

20 

20 

ATC_MD 

t40 

1 

1 

5 

5 

ATC_MD 

t41 

1 

1 

20 

20 

ATC  MD 
YPG_AZ 

t53 

1 

1 

15 

15 

WSMRJMM 

t56 

1 

1 

22 

22 

ATC  MD 
AVTB  CA 
WSMR  NM 
YPG_AZ 

t57 

1 

1 

22 

22 

ATC  MD 
AVTB  CA 
WSMR  NM 
YPG_AZ 

t58 

1 

1 

22 

22 

ATC  MD 
AVTB  CA 
WSMR  NM 
YPG_AZ 

t59 

1 

1 

22 

22 

ATC  MD 
AVTB  CA 
WSMR  NM 
YPG  AZ 

Actual 

Location 

Successor 

Successor 

Success? 

Priority 

Priority 

Notes 

ATC 

high 

AVTB 

high 

ATC 

medium 

ATC 

t39 

Yes 

medium 

performed 
before 
high  tasks 

ATC 

medium 

WSMR 

t22 

Yes 

medium 

AVTB 

high 

ATC 

high 

AVTB  high 


ATC  high  should 

have  been 
performed 
on  a 
different 
vehicle 
and  earlier 
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Event 

Test 

Asset 

Input 

Actus 

Test 

Asset 

tl  Test 

Period 
s  Input 

Actual 
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Input 

Location 

Actual 

Location 

Successor 

Successor 

Success? 

Priority 

Priority 

Notes 

t60 
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1 

22 

22 

ATC  MD 
AVTB  CA 
WSMR  NM 
YPG_AZ 

WSMR 

medium 

performed 
before 
high  tasks 

t61 

1 

1 

22 

22 

ATC  MD 
AVTB  CA 
WSMR  NM 
YPG_AZ 

AVTB 

medium 

t62 

1 

1 

22 

22 

ATC  MD 
AVTB  CA 
WSMR  NM 
YPG_AZ 

AVTB 

medium 

t63 

1 

1 

21 

21 

AVTB_CA 

AVTB 

medium 

t64 

1 

1 

15 

15 

AT  C_M  D 

ATC 

high 
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Using  the  information  provided  in  the  t_periods  schedule  and  the 
comparison  table,  the  verification  assessment  results  for  t_periods  schedule 
generated  by  the  model  is  given  in  the  Table  47  in  the  “Verification  &  Rationale” 
column,  which  is  relative  to  Table  4  requirements  that  are  repeated  here. 


Table  47.  Model  Controls  and  Constraints  t_periods  Schedule 

Verification 


Para  # 

Para  Title 

Requirement 

Verification  Criteria 

Verification  & 
Rationale 

2.0 

Controls  and 
Constraints 

Not  Applicable 

Not  Applicable 

Not  Applicable 

2.1 

Test  Period 
(T=0) 

The  TE  RSCSP 
model  schedule 
shall  be 
constrained  by 
the  test  period. 

The  requirement  shall  be 
verified  by  confirming  that 
the  schedule  is  within  the 
test  period  given  in  the  test 
period  input  file. 

MET 

Test  period  is  180 
and  model  test 
period  used  is  66 
days. 

2.1.1 

One  Test  Event 
on  Test  Asset 
during  Time 
Period  (T=0) 

The  TE  RSCSP 
model  shall  not 
allow  more  than 
one  test  event 
during  a  time 
period  on  a  test 
asset. 

The  requirement  shall  be 
verified  by  confirming  that 
the  schedule  does  not 
show  more  than  one  test 
event  in  a  time  period  on  a 
test  asset. 

PARTIALLY  MET 
Output  File  does  not 
assign  a  test  asset, 
but  in  aggregate 
does  not  exceed  the 
number  of  test 
assets. 

2.1.2 

Place  all  Test 
Events  in  Test 
Period  (T=0) 

The  TE  RSCSP 
model  shall 
place  all 
identified  test 
events  in  a  test 
period. 

The  requirement  shall  be 
verified  by  confirming  that 
schedule  places  all 
identified  test  events  in  a 
test  period. 

MET 

Actual  test  events 
from  input  file  are  all 
placed  in  test 
periods. 

2.1.3 

Test  Event  Test 
Period 

Durations 

(T=0) 

The  TE  RSCSP 
model  shall  use 
the  test  event 
test  period 
durations  input. 

The  requirement  shall  be 
verified  by  confirming  that 
schedule  test  event  test 
period  durations  matches 
the  test  events  test  period 
input  file. 

MET 

Actual  test  event 
test  periods  tracks  to 
input  tile. 

2.2 

Test  Events 
(T=0) 

The  TE  RSCSP 
model  schedule 
shall  be 
constrained  by 
the  test  events. 

The  requirement  shall  be 
verified  by  confirming  that 
the  schedule  includes  all 
test  events  included  in  the 
test  events  input  file. 

MET 

Actual  test  events 
used  track  to  input 
file. 

2.3 

Test  Asset 

Type  (0) 

The  TE  RSCSP 
model  shall  use 
test  asset  type 
inputs. 

The  requirement  shall  be 
verified  by  confirming  that 
schedule  results  places 
test  events  on  the 

NOT  VERIFIED 

Input  files  did  not 
use  more  than  one 
test  asset  type. 

applicable  test  asset  type 
based  on  the  input  file. 
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Para  # 

Para  Title 

Requirement 

Verification  Criteria 

Verification  & 
Rationale 

2.4 

Test  Venues 
(T=0) 

The  TE  RSCSP 
model  shall  use 
test  venues 
input. 

The  requirement  shall  be 
verified  by  confirming  that 
the  schedule  includes  only 
the  test  venues  included  in 
the  test  venues  input  file. 

MET 

Actual  test  venues 
used  track  to  input 
file. 

2.5 

Test  Event 
Priorities 

Not  Applicable 

Not  Applicable 

Not  Applicable 

2.5.1 

Test  Event 
Priority 
Relationships 
(T=0) 

The  TE  RSCSP 
model  shall  be 
constrained  by 
the  test  event 
priority 
relationships 

The  requirement  shall  be 
verified  by  confirming  that 
schedule  results  are 
constrained  by  the  test 
event  priority  relationships 
input  file. 

PARTIALLY  MET 

In  most  cases,  MET. 
There  is  an  issue 
with  t07,  tl  1 ,  t22, 
t24,  t40,  and  t59 
medium  test  events 
happening  before 
high  priority  tests. 

2.5.2 

High  to  Medium 
Priority 
Relationship 
(T=0) 

The  TE  RSCSP 
model  high 
priority  test 
events  shall  be 
started  before 
medium  priority 
test  events. 

The  requirement  shall  be 
verified  by  confirming  that 
schedule  starts  high 
priority  test  events  before 
medium  priority  test 
events. 

PARTIALLY  MET 

In  most  cases,  MET. 
There  is  an  issue 
with  t07,  tl  1 ,  t22, 
t24,  t40,  and  t59 
medium  happening 
before  high  priority 
tests. 

2.5.3 

Medium  to  Low 
Priority 
Relationship 
(T=0) 

The  TE  RSCSP 
model  medium 
priority  test 
events  shall  be 
started  before 
low  priority  test 
events. 

The  requirement  shall  be 
verified  by  confirming  that 
schedule  starts  medium 
priority  test  events  before 
low  priority  test  events. 

NOT  VERIFIED 

Model  run  did  not 
include  any  low 
priority  test  events. 

2.5.4 

High  to  Low 
Priority 
Relationships 
(T=0) 

The  TE  RSCSP 
model  high 
priority  test 
events  shall  be 
started  before 
low  priority  test 
events. 

The  requirement  shall  be 
verified  by  confirming  that 
schedule  starts  high 
priority  test  events  before 
low  priority  test  events. 

NOT  VERIFIED 

Model  run  did  not 
include  any  low 
priority  test  events. 

2.6 

Test  Venue 
Movement  Test 
Periods  (T=0) 

The  TE  RSCSP 
model  shall  be 
constrained  by 
the  test  venue 
movement  test 
periods. 

The  requirement  shall  be 
verified  by  confirming  that 
schedule  uses  the  test 
venue  movement  test 
periods  based  on  the  input 
file. 

MET 

Data  shows  that 
when  moving 
venues,  the  input 
data  matches  the 
input  file. 

2.6.1 

Add  Time 

Period  from 

Test  Venue 

Movement 

(T=0) 

The  TE  RSCSP 
model  shall  add 
time  periods  to 
the  schedule 
based  on  the 
distance 
between  test 
venues. 

The  requirement  shall  be 
verified  by  confirming  that 
schedule  results  show  that 
the  time  periods  added  to 
the  schedule  when  the  test 
asset  moves  to  a  new  test 
venue  are  based  on  the 
input  file. 

MET 

Data  shows  that 
when  moving 
venues,  the  input  file 
test  periods  are 
added  to  the  test 
schedule. 
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Para  # 

Para  Title 

Requirement 

Verification  Criteria 

Verification  & 
Rationale 

2.6.2 

Schedule  Test 
Event  on 
Available  Test 
Venues  (T=0) 

The  TE  RSCSP 
model  shall  not 
schedule  a  test 
event  at  a  test 
venue  that  does 
not  perform  that 
test  event. 

The  requirement  shall  be 
verified  by  confirming  that 
schedule  results  show  that 
the  test  events  are 
scheduled  only  on 
available  test  venues 
based  on  the  input  file. 

MET 

Test  venues  used 
track  to  test  venues 
allowed  based  on 
the  input  file. 

2.7 

Test  Venue 

Test  Asset 
Capacity  (T=0) 

The  TE  RSCSP 
model  shall  be 
constrained  by 
test  venue  test 
asset  capacity 
input. 

The  requirement  shall  be 
verified  by  confirming  that 
the  number  of  test  assets 
located  at  each  test  venue 
on  the  schedule  does  not 
exceed  the  capacity 
identified  in  the  input  file. 

NOT  VERIFIED 

The  number  of  test 
assets  assigned  to  a 
test  venue  by  the 
model  did  not 
exceed  the  amount 
in  the  file,  but  the 
numbers  in  the  file 
(10)  did  not  really 
check  the  model. 

2.8 

Test  Asset  Test 
Venue  Starting 
Location  (T=0) 

The  TE  RSCSP 
model  shall  be 
constrained  by 
the  test  asset 
test  venue 
starting  location 
input. 

The  requirement  shall  be 
verified  by  confirming  that 
schedule  test  assets  start 
at  the  venues  identified  in 
the  test  asset  test  venue 
input  file. 

MET 

The  starting 
locations  and 
quantities  (4  at  ATC 
and  3  at  AVTB) 
track  to  the  input  file. 

2.9 

Test  Event  Test 
Venue 

Relationships 

(T=0) 

The  TE  RSCSP 
model  shall  be 
constrained  by 
the  test  event 
test  venue 
relationships 
input. 

The  requirement  shall  be 
verified  by  confirming  that 
schedule  test  events  are 
performed  at  the  test 
venues  identified  in  the 
test  event  test  venue 
relationships  input  file. 

MET 

Test  events  are 
assigned  to  allowed 
test  venues  based 
on  the  input  file. 

2.10 

Test  Event 
Precedence 
Relationships 
(T=0) 

The  TE  RSCSP 
model  shall  be 
constrained  by 
test  event 
precedence 
relationships 
input. 

The  requirement  shall  be 
verified  by  confirming  that 
schedule  results  are 
constrained  by  the  test 
event  precedence 
relationships  input  file. 

MET 

Data  shows  that 
predecessor  test 
events  in  the 
schedule  occur 
before  successor 
test  events. 

2.10.1 

Test  Event 
Predecessor 

First  (T=0) 

The  TE  RSCSP 
model  shall 
perform 

predecessor  test 
events  prior  to 
test  event 
successor  test 
events. 

The  requirement  shall  be 
verified  by  confirming  that 
predecessor  test  events 
are  on  the  schedule  before 
successor  test  events 
regardless  of  test  event 
priority. 

MET 

Data  shows  that 
predecessor  test 
events  in  the 
schedule  occur 
before  successor 
test  events. 
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Para  #  Para  Title 


Requirement 


Verification  Criteria 


Verification  & 
Rationale 


2.10.2 

Test  Event 
Successor 
Second (T=0) 

The  TE  RSCSP 
model  shall  not 
perform 
successor  test 
events  prior  to 
test  event 
predecessor  test 
events. 

The  requirement  shall  be 
verified  by  confirming  that 
successor  test  events  are 
on  the  schedule  before 
predecessor  test  events 
regardless  of  test  event 
priority. 

MET 

Data  shows  that 
successor  test 
events  in  the 
schedule  occur  after 
predecessor  test 
events. 

2.11 

Number  of  Test 
Assets  Needed 
for  Test  Events 
(T=0) 

The  TE  RSCSP 
model  shall  use 
the  number  of 
test  assets 
needed  for  test 
events  input. 

The  requirement  shall  be 
verified  by  confirming  that 
the  schedule  uses  the 
number  of  test  assets 
needed  for  test  events 
input  file. 

MET 

Data  shows  that  the 
number  of  test 
assets  needed  for  a 
test  event  are  used. 
For  multi-vehicle 
operations,  tests  are 
performed 
simultaneously  on 
the  test  assets. 

2.12 

Low  Priority 
Schedule 
Relationship 
(0) 

The  TE  RSCSP 
model  low 
priority  test 
events  shall  be 
allowed  to  go 
beyond  the  test 
period. 

The  requirement  shall  be 
verified  by  confirming  that 
schedule  results  match  the 
input  file. 

NOT  VERIFIED 

Low  priority  test 
events  are  not  used 
in  this  model  run. 

2.13 

Test  Asset 
Unavailability 
(0) 

The  TE  RSCSP 
model  shall  be 
constrained  by 
the  test  asset 
unavailability. 

The  requirement  shall  be 
verified  by  confirming  that 
schedule  places  test 
assets  only  on  available 
test  assets  based  on  the 
input  file. 

NOT  VERIFIED 

Data  for  Test  Asset 
Unavailability  is  not 
included  in  the 
model  run. 

2.14 

Test  Venue 
Unavailability 
(0) 

The  TE  RSCSP 
model  shall  be 
constrained  by 
test  venue 
unavailability. 

The  requirement  shall  be 
verified  by  confirming  that 
schedule  results  are 
constrained  by  the  input 
file. 

NOT  VERIFIED 

Data  for  Test  Venue 
Unavailability  is  not 
included  in  the 
model  run. 

2.15 

Test  Event 
Priority 

Deadlines 

(T=0) 

The  TE  RSCSP 
model  shall  be 
constrained  by 
the  high  and 
medium  test 
event  priority 
deadlines. 

The  requirement  shall  be 
verified  by  confirming  that 
schedule  results  are 
constrained  by  the  input 
file. 

PARTIALLY  MET 
This  model  run  used 

1 90  for  the  total  test 
periods.  High  is  170, 
and  medium  is  190. 
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Para  # 

Para  Title 

Requirement 

Verification  Criteria 

Verification  & 
Rationale 

2.15.1 

High  Priority  to 
Deadline 
Relationship 
(T-O) 

The  TE  RSCSP 
model  shall 
complete  high 
priority  test 
events  before 
high  priority 
deadlines. 

The  requirement  shall  be 
verified  by  confirming  that 
schedule  results  start  high 
priority  test  events  before 
the  high  priority  deadline. 

MET 

This  model  run  used 

1 90  for  the  total  test 
periods.  High  test 
periods  is  170,  and 
there  are  no  high 
priority  test  events 
that  go  beyond  this 
deadline.  Schedule 
ends  at  test  period 

66. 

2.15.2 

Medium  Priority 
to  Deadline 
Relationship 
(T=0) 

The  TE  RSCSP 
model  shall 
complete 
medium  priority 
test  events 
before  medium 
priority 
deadlines. 

The  requirement  shall  be 
verified  by  confirming  that 
schedule  results  start 
medium  priority  test  events 
before  the  medium  priority 
deadline. 

NOT  VERIFIED 

This  model  run  used 

1 90  for  the  total  test 
periods  and  the 
medium  test 
periods,  which 
means  that  it  is  not 
verified. 

2.15.3 

High  Priority  to 
Test  Period 
Relationship 
(T=0) 

The  TE  RSCSP 
model  shall 
complete  high 
priority  test 
events  before 
the  test  period 
completes. 

The  requirement  shall  be 
verified  by  confirming  that 
schedule  results  complete 
high  priority  test  events 
before  the  test  period 
completes. 

MET 

High  priority  test 
events  end  at  66. 
Schedule  ends  at 
test  period  66.  Test 
period  ends  at  190. 

2.15.4 

Medium  Priority 
to  Test  Period 
Relationship 
(T=0) 

The  TE  RSCSP 
model  shall 
complete 
medium  priority 
test  events 
before  the  test 
period 
completes. 

The  requirement  shall  be 
verified  by  confirming  that 
schedule  results  complete 
medium  priority  test  events 
before  the  test  period 
completes. 

MET 

Medium  priority  test 
events  end  at  test 
period  66.  Schedule 
ends  at  test  period 

66. 

Test  period  ends  at 
190. 

The  results  of  the  model  controls  and  constraints  requirements  schedule 
verification  assessment  for  t_periods  indicate  that  the  majority  of  the  threshold 
model  controls  and  constraints  requirements  in  Table  4  are  MET,  except  for 
some  that  are  PARTIALLY  MET  and  some  that  are  NOT  VERIFIED.  There  are 
no  controls  and  constraints  requirements  that  are  verified  to  be  NOT  MET. 

Partially  Met  requirements.  Data  used  for  verification  for  time  period, 
priority  deadlines  did  not  fully  test  the  time  period  due  to  the  values  used.  The 
following  are  considered  to  be  PARTIALLY  MET,  and  require  an  additional 
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verification  event  in  order  to  be  fully  assessed:  One  Test  Event  on  Test  Asset 
during  Time  Period,  Test  Event  Priority  Relationships,  High  to  Medium  Priority 
Relationship,  and  Test  Event  Priority  Deadlines. 

Not  Verified  requirements.  Data  did  not  include  low  priority  test  events, 
test  asset  type,  test  venue  capacity,  test  asset  unavailability,  test  venue 
unavailability,  and  medium  deadline.  The  following  are  considered  to  be  NOT 
VERIFIED,  and  require  an  additional  verification  event  in  order  to  be  assessed: 
Medium  to  Low  Priority  Relationship,  High  to  Low  Priority  Relationships,  Test 
Venue  Test  Asset  Capacity,  Test  Asset  Type  Test  Event  Relationships,  Test 
Asset  Unavailability,  Test  Venue  Unavailability,  Medium  Priority  to  Deadline 
Relationship,  and  Low  Priority  Schedule  Relationship. 
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Figure  33.  First  Month  of  Model  Beta_min  Detailed  Schedule 
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Figure  34.  Second  Month  of  Model  Betajmin  Detailed  Schedule 
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Figure  35.  Third  Month  of  Model  Beta_min  Detailed  Schedule 
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Figure  36.  Third  Month  of  Model  Betajnin  Detailed  Schedule 
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Table  48.  Model  Input  Files  to  Model  Beta_min  Detailed  Schedule  Comparison 


Test 

Event 

Test 

Asset 

Input 

Actual 

Test 

Assets 

Test 

Period 

Input 

Actual 

Test 

Periods 

Input 

Location 

Actual 

Locati 

on 

Successor 

Successor 

Success? 

Priority 

Priority 

Issues 

toi 

1 

1 

1 

1 

ATC_MD 

ATC 

t02 

Yes 

high 

t02 

1 

1 

3 

3 

AT  C_M  D 

ATC 

high 

t03 

1 

1 

6 

6 

ATC_MD 

ATC 

high 

t04 

1 

1 

3 

3 

AT  C_M  D 

ATC 

high 

t05 

1 

1 

3 

3 

ATC_MD 

ATC 

high 

t06 

1 

1 

3 

3 

AT  C_M  D 

ATC 

high 

t07 

2 

2 

3 

3 

ATC_MD 

ATC 

medium 

t08 

1 

1 

4 

4 

ATC_MD 

ATC 

high 

t09 

1 

1 

14 

14 

AT  C_M  D 

ATC 

medium 

tio 

1 

1 

3 

3 

ATC_MD 

ATC 

high 

til 

1 

1 

3 

3 

ATC_MD 

ATC 

t09 

Yes 

medium 

performed 
before  high 
tasks 

t12 

1 

1 

6 

6 

AT  C_M  D 

ATC 

high 

t13 

3 

3 

13 

13 

AVTBCA 

AVTB 

tl  4,  tl  5,  tl  6,  tl  7, 
tl  8,  tl  9,  t20,  t22, 
t23,  t24,  t63 

Yes 

high 

t14 

1 

1 

7 

7 

AVTB_CA 

AVTB 

high 
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Test  Test  Actual 

Event  Asset  Test 

Input  Assets 

tis 

116 

t17 

ti8 

t19 

t20 

t22 

123  V] 

t24 

t25  HMfe— M 


t27 

t28 

t29 

t33 


Actual  Input 

Test  Location 

Periods 


3  AVTB_CA 
3  AVTBCA 
3  AVTB_CA 
1  AVTBCA 
1  AVTB_CA 

3  AVTBCA 

4  AVTB_CA 
4  AVTBCA 
13  ATC  MD 


1  ATC_MD 
7  ATC_MD 
7  ATC_MD 
3  AVTB  CA 


3 

3 

3 

3 

1 

1 

3 

4 

4 

13 

1 

7 

7 

3 


Actual 

Successor 

Successor 

Priority 

Priority 

Locati 

Success? 

Issues 

on 

AVTB 

ti  6,  ti  7,  ti  8 

Yes 

high 

AVTB 

high 

AVTB 

high 

AVTB 

high 

AVTB 

high 

AVTB 

high 

AVTB 

high 

AVTB 

medium 

AVTB 

medium 

tOI,  t02,  t03,  t04, 
t05,  t06,  t07,  t08, 
t09,  ti 0,  til,  ti 2, 
ti  4,  ti  5,  ti  6,  ti  7, 
ti  8,  ti  9,  t20,  t22, 
t23,  t24,  t27,  t28, 
t29,  t33,  t36,  t39 
t40,  t41 ,  t53,  t64 

yes 

high 

ATC 

high 

ATC 

high 

ATC 

high 

AVTB 

high 
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Test 

Event 

Test 

Asset 

Input 

Actual 

Test 

Assets 

Test 

Period 

Input 

Actual 

Test 

Periods 

Input 

Location 

t36 

1 

1 

7 

7 

ATC_MD 

t37 

1 

1 

7 

7 

AVTB_CA 

t39 

1 

1 

29 

29 

ATC_MD 

t40 

1 

1 

7 

7 

ATC_MD 

t41 

1 

1 

29 

29 

ATC  MD 
YPG_AZ 

t53 

1 

1 

21 

21 

WSMRJMM 

t56 

1 

1 

22 

22 

ATC  MD 
AVTB  CA 
WSMR  NM 
YPG  AZ 

t57 

1 

1 

22 

22 

ATC  MD 
AVTB  CA 
WSMR  NM 
YPG_AZ 

t58 

1 

1 

22 

22 

ATC  MD 
AVTB  CA 
WSMR  NM 
YPG_AZ 

t59 

1 

1 

22 

22 

ATC  MD 
AVTB  CA 
WSMR  NM 
YPG  AZ 

Actual 

Locati 

on 

Successor 

Successor 

Success? 

Priority 

Priority 

Issues 

ATC 

high 

AVTB 

high 

ATC 

medium 

ATC 

t39 

Yes 

medium 

performed 
before  high 
tasks 

ATC 

medium 

WSM 

R 

t22 

Yes 

medium 

performed 
before  high 
tasks 

AVTB 

high 

ATC 

high 

ATC 


high 


ATC 


high 
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Test 

Event 

Test 

Asset 

Input 

Actual 

Test 

Assets 

Test 

Period 

Input 

Actual 

Test 

Periods 

Input 

Location 

t60 

1 

1 

22 

22 

ATC  MD 
AVTB  CA 
WSMR  NM 
YPG_AZ 

t61 

1 

1 

22 

22 

ATC  MD 
AVTB  CA 
WSMR  NM 
YPG_AZ 

t62 

1 

1 

22 

22 

ATC  MD 
AVTB  CA 
WSMR  NM 
YPG_AZ 

t63 

1 

1 

21 

21 

AVTB_CA 

t64 

1 

1 

15 

15 

ATC_MD 

Actual 

Locati 

on 

Successor 

Successor 

Success? 

Priority 

Priority 

Issues 

ATC 

medium 

AVTB  medium 


WSM 

R 


AVTB 


medium  performed 

before  high 
tasks 
probably 
due  to 
venue 

medium 


ATC 


high 
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Using  the  information  provided  in  the  Beta_min  schedule  and  the 
comparison  table,  the  verification  assessment  results  for  Betajmin  schedule 
generated  by  the  model  is  given  in  the  Table  49  in  the  “Verification  &  Rationale” 
column,  which  is  relative  to  Table  4  requirements  that  are  repeated  here. 

Table  49.  Model  Controls  and  Constraints  Betajmin  Schedule 

Verification 


Para  # 

Para  Title 

Requirement 

Verification  Criteria 

Verification  & 
Rationale 

2.0 

Controls  and 
Constraints 

Not  Applicable 

Not  Applicable 

Not  Applicable 

2.1 

Test  Period 
(T=0) 

The  TE  RSCSP 
model  schedule 
shall  be 

constrained  by  the 
test  period. 

The  requirement  shall  be 
verified  by  confirming 
that  the  schedule  is 
within  the  test  period 
given  in  the  test  period 
input  file. 

MET 

Test  period  is  180 
and  model  test 
period  used  is  66 
days. 

2.1.1 

One  Test  Event 
on  Test  Asset 
during  Time 
Period  (T=0) 

The  TE  RSCSP 
model  shall  not 
allow  more  than 
one  test  event 
during  a  time 
period  on  a  test 
asset. 

The  requirement  shall  be 
verified  by  confirming 
that  the  schedule  does 
not  show  more  than  one 
test  event  in  a  time 
period  on  a  test  asset. 

PARTIALLY  MET 
Output  File  does 
not  assign  a  test 
asset,  but  in 
aggregate  does  not 
exceed  the  number 
of  test  assets. 

2.1.2 

Place  all  Test 
Events  in  Test 
Period  (T=0) 

The  TE  RSCSP 
model  shall  place 
all  identified  test 
events  in  a  test 
period. 

The  requirement  shall  be 
verified  by  confirming 
that  schedule  places  all 
identified  test  events  in  a 
test  period. 

MET 

Actual  test  events 
from  input  file  are 
all  placed  in  test 
periods. 

2.1.3 

Test  Event  Test 
Period  Durations 
(T=0) 

The  TE  RSCSP 
model  shall  use 
the  test  event  test 
period  durations 
input. 

The  requirement  shall  be 
verified  by  confirming 
that  schedule  test  event 
test  period  durations 
matches  the  test  events 
test  period  input  file. 

MET 

Actual  test  event 
test  periods  tracks 
to  input  tile. 

2.2 

Test  Events 
(T=0) 

The  TE  RSCSP 
model  schedule 
shall  be 

constrained  by  the 
test  events. 

The  requirement  shall  be 
verified  by  confirming 
that  the  schedule 
includes  all  test  events 
included  in  the  test 
events  input  file. 

MET 

Actual  test  events 
used  track  to  input 
file. 
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Para  # 

Para  Title 

Requirement 

Verification  Criteria 

Verification  & 
Rationale 

2.3 

Test  Asset  Type 
(0) 

The  TE  RSCSP 
model  shall  use 
test  asset  type 
inputs. 

The  requirement  shall  be 
verified  by  confirming 
that  schedule  results 
places  test  events  on  the 
applicable  test  asset 
type  based  on  the  input 
file. 

NOT  VERIFIED 

Input  files  did  not 
use  more  than  one 
test  asset  type. 

2.4 

Test  Venues 
(T=0) 

The  TE  RSCSP 
model  shall  use 
test  venues 
input. 

The  requirement  shall  be 
verified  by  confirming 
that  the  schedule 
includes  only  the  test 
venues  included  in  the 
test  venues  input  file. 

MET 

Actual  test  venues 
used  track  to  input 
file. 

2.5 

Test  Event 
Priorities 

Not  Applicable 

Not  Applicable 

Not  Applicable 

2.5.1 

Test  Event 

Priority 

Relationships 

(T=0) 

The  TE  RSCSP 
model  shall  be 
constrained  by 
the  test  event 
priority 
relationships 

The  requirement  shall  be 
verified  by  confirming 
that  schedule  results  are 
constrained  by  the  test 
event  priority 
relationships  input  file. 

PARTIALLY  MET 

In  most  cases, 

MET.  There  is  an 
issue  with  t22,  t40, 
and  t53  medium 
test  events 
happening  before 
high  priority  tests. 

2.5.2 

High  to  Medium 
Priority 
Relationship 
(T=0) 

The  TE  RSCSP 
model  high 
priority  test 
events  shall  be 
started  before 
medium  priority 
test  events. 

The  requirement  shall  be 
verified  by  confirming 
that  schedule  starts  high 
priority  test  events 
before  medium  priority 
test  events. 

PARTIALLY  MET 

In  most  cases, 

MET.  There  is  an 
issue  with  t22,  t40, 
and  t53  medium 
happening  before 
high  priority  tests. 

2.5.3 

Medium  to  Low 
Priority 
Relationship 
(T=0) 

The  TE  RSCSP 
model  medium 
priority  test 
events  shall  be 
started  before 
low  priority  test 
events. 

The  requirement  shall  be 
verified  by  confirming 
that  schedule  starts 
medium  priority  test 
events  before  low  priority 
test  events. 

NOT  VERIFIED 
Model  run  did  not 
include  any  low 
priority  test  events. 

2.5.4 

High  to  Low 
Priority 
Relationships 
(T=0) 

The  TE  RSCSP 
model  high 
priority  test 
events  shall  be 
started  before 
low  priority  test 
events. 

The  requirement  shall  be 
verified  by  confirming 
that  schedule  starts  high 
priority  test  events 
before  low  priority  test 
events. 

NOT  VERIFIED 
Model  run  did  not 
include  any  low 
priority  test  events. 

2.6 

Test  Venue 
Movement  Test 
Periods  (T=0) 

The  TE  RSCSP 
model  shall  be 
constrained  by 
the  test  venue 
movement  test 
periods. 

The  requirement  shall  be 
verified  by  confirming 
that  schedule  uses  the 
test  venue  movement 
test  periods  based  on 
the  input  file. 

MET 

Data  shows  that 
when  moving 
venues,  the  input 
data  matches  the 
input  file. 
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Para  # 

Para  Title 

Requirement 

Verification  Criteria 

Verification  & 
Rationale 

2.6.1 

Add  Time  Period 
from  Test  Venue 
Movement  (T=0) 

The  TE  RSCSP 
model  shall  add 
time  periods  to 
the  schedule 
based  on  the 
distance  between 
test  venues. 

The  requirement  shall  be 
verified  by  confirming 
that  schedule  results 
show  that  the  time 
periods  added  to  the 
schedule  when  the  test 
asset  moves  to  a  new 
test  venue  are  based  on 
the  input  file. 

MET 

Data  shows  that 
when  moving 
venues,  the  input 
file  test  periods  are 
added  to  the  test 
schedule. 

2.6.2 

Schedule  Test 
Event  on 

Available  Test 
Venues  (T=0) 

TheTE  RSCSP 
model  shall  not 
schedule  a  test 
event  at  a  test 
venue  that  does 
not  perform  that 
test  event. 

The  requirement  shall  be 
verified  by  confirming 
that  schedule  results 
show  that  the  test  events 
are  scheduled  only  on 
available  test  venues 
based  on  the  input  file. 

MET 

Test  venues  used 
track  to  test  venues 
allowed  based  on 
the  input  file. 

2.7 

Test  Venue  Test 
Asset  Capacity 
(T=0) 

TheTE  RSCSP 
model  shall  be 
constrained  by 
test  venue  test 
asset  capacity 
input. 

The  requirement  shall  be 
verified  by  confirming 
that  the  number  of  test 
assets  located  at  each 
test  venue  on  the 
schedule  does  not 
exceed  the  capacity 
identified  in  the  input  file. 

NOT  VERIFIED 

The  number  of  test 
assets  assigned  to 
a  test  venue  by  the 
model  did  not 
exceed  the  amount 
in  the  file,  but  the 
numbers  in  the  file 
(10)  did  not  really 
check  the  model. 

2.8 

Test  Asset  Test 
Venue  Starting 
Location  (T=0) 

TheTE  RSCSP 
model  shall  be 
constrained  by 
the  test  asset  test 
venue  starting 
location  input. 

The  requirement  shall  be 
verified  by  confirming 
that  schedule  test  assets 
start  at  the  venues 
identified  in  the  test 
asset  test  venue  input 
file. 

MET 

The  starting 
locations  and 
quantities  (4  at 

ATC  and  3  at 

AVTB)  track  to  the 
input  file. 

2.9 

Test  Event  Test 
Venue 

Relationships 

(T=0) 

The  TE  RSCSP 
model  shall  be 
constrained  by 
the  test  event  test 

venue 

relationships 

input. 

The  requirement  shall  be 
verified  by  confirming 
that  schedule  test  events 
are  performed  at  the  test 
venues  identified  in  the 
test  event  test  venue 
relationships  input  file. 

MET 

Test  events  are 
assigned  to  allowed 
test  venues  based 
on  the  input  file. 

2.10 

Test  Event 
Precedence 
Relationships 
(T=0) 

TheTE  RSCSP 
model  shall  be 
constrained  by 
test  event 
precedence 
relationships 
input. 

The  requirement  shall  be 
verified  by  confirming 
that  schedule  results  are 
constrained  by  the  test 
event  precedence 
relationships  input  file. 

MET 

Data  shows  that 
predecessor  test 
events  in  the 
schedule  occur 
before  successor 
test  events. 
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Para  #  Para  Title  Requirement  Verification  Criteria  Verification  & 


2.10.1 

Test  Event 
Predecessor  First 
(T=0) 

The  TE  RSCSP 
model  shall 
perform 

predecessor  test 
events  prior  to 
test  event 
successor  test 
events. 

The  requirement  shall  be 
verified  by  confirming 
that  predecessor  test 
events  are  on  the 
schedule  before 
successor  test  events 
regardless  of  test  event 
priority. 

MET 

Data  shows  that 
predecessor  test 
events  in  the 
schedule  occur 
before  successor 
test  events. 

2.10.2 

Test  Event 
Successor 

Second  (T=0) 

TheTE  RSCSP 
model  shall  not 
perform 
successor  test 
events  prior  to 
test  event 
predecessor  test 
events. 

The  requirement  shall  be 
verified  by  confirming 
that  successor  test 
events  are  on  the 
schedule  before 
predecessor  test  events 
regardless  of  test  event 
priority. 

MET 

Data  shows  that 
successor  test 
events  in  the 
schedule  occur 
after  predecessor 
test  events. 

2.11 

Number  of  Test 
Assets  Needed 
for  Test  Events 
(T=0) 

The  TE  RSCSP 
model  shall  use 
the  number  of 
test  assets 
needed  for  test 
events  input. 

The  requirement  shall  be 
verified  by  confirming 
that  the  schedule  uses 
the  number  of  test 
assets  needed  for  test 
events  input  file. 

MET 

Data  shows  that 
the  number  of  test 
assets  needed  for  a 
test  event  are  used. 
For  multi-vehicle 
operations,  tests 
are  performed 
simultaneously  on 
the  test  assets. 

2.12 

Low  Priority 
Schedule 
Relationship  (0) 

TheTE  RSCSP 
model  low  priority 
test  events  shall 
be  allowed  to  go 
beyond  the  test 
period. 

The  requirement  shall  be 
verified  by  confirming 
that  schedule  results 
match  the  input  file. 

NOT  VERIFIED 

Low  priority  test 
events  are  not  used 
in  this  model  run. 

2.13 

Test  Asset 
Unavailability  (0) 

The  TE  RSCSP 
model  shall  be 
constrained  by 
the  test  asset 
unavailability. 

The  requirement  shall  be 
verified  by  confirming 
that  schedule  places  test 
assets  only  on  available 
test  assets  based  on  the 
input  file. 

NOT  VERIFIED 

Data  for  Test  Asset 
Unavailability  is  not 
included  in  the 
model  run. 

2.14 

Test  Venue 
Unavailability  (0) 

TheTE  RSCSP 
model  shall  be 
constrained  by 
test  venue 
unavailability. 

The  requirement  shall  be 
verified  by  confirming 
that  schedule  results  are 
constrained  by  the  input 
file. 

NOT  VERIFIED 

Data  for  Test 

Venue 

Unavailability  is  not 
included  in  the 
model  run. 

2.15 

Test  Event 

Priority  Deadlines 
(T=0) 

The  TE  RSCSP 
model  shall  be 
constrained  by 
the  high  and 
medium  test 
event  priority 
deadlines. 

The  requirement  shall  be 
verified  by  confirming 
that  schedule  results  are 
constrained  by  the  input 
file. 

PARTIALLY  MET 
This  model  run 
used  190  for  the 
total  test  periods. 

High  is  170,  and 
medium  is  190. 
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Para  # 

Para  Title 

Requirement 

Verification  Criteria 

Verification  & 
Rationale 

2.15.1 

High  Priority  to 
Deadline 
Relationship 
(T=0) 

TheTE  RSCSP 
model  shall 
complete  high 
priority  test 
events  before 
high  priority 
deadlines. 

The  requirement  shall  be 
verified  by  confirming 
that  schedule  results 
start  high  priority  test 
events  before  the  high 
priority  deadline. 

MET 

This  model  run 
used  190  for  the 
total  test  periods. 

High  test  periods  is 
170,  and  there  are 
no  high  priority  test 
events  that  go 
beyond  this 
deadline.  Schedule 
ends  at  test  period 

80. 

2.15.2 

Medium  Priority 
to  Deadline 
Relationship 
(T=0) 

The  TE  RSCSP 
model  shall 
complete  medium 
priority  test 
events  before 
medium  priority 
deadlines. 

The  requirement  shall  be 
verified  by  confirming 
that  schedule  results 
start  medium  priority  test 
events  before  the 
medium  priority 
deadline. 

NOT  VERIFIED 

This  model  run 
used  190  for  the 
total  test  periods 
and  the  medium 
test  periods,  which 
means  that  it  is  not 
verified. 

2.15.3 

High  Priority  to 
Test  Period 
Relationship 
(T=0) 

TheTE  RSCSP 
model  shall 
complete  high 
priority  test 
events  before  the 
test  period 
completes. 

The  requirement  shall  be 
verified  by  confirming 
that  schedule  results 
complete  high  priority 
test  events  before  the 
test  period  completes. 

MET 

High  priority  test 
events  end  at  53. 
Schedule  ends  at 
test  period  80.  Test 
period  ends  at  190. 

2.15.4 

Medium  Priority 
to  Test  Period 
Relationship 
(T=0) 

The  TE  RSCSP 
model  shall 
complete  medium 
priority  test 
events  before  the 
test  period 
completes. 

The  requirement  shall  be 
verified  by  confirming 
that  schedule  results 
complete  medium 
priority  test  events 
before  the  test  period 
completes. 

MET 

Medium  priority  test 
events  end  at  test 
period  53. 

Schedule  ends  at 
test  period  80. 

Test  period  ends  at 
190. 

The  results  of  the  model  controls  and  constraints  requirements  schedule 
verification  assessment  for  Beta_min  indicate  that  the  majority  of  the  threshold 
model  controls  and  constraints  requirements  in  Table  49  are  MET,  except  for 
some  that  are  PARTIALLY  MET  and  some  that  are  NOT  VERIFIED.  There  are 
no  controls  and  constraints  requirements  that  are  verified  to  be  NOT  MET. 

Partially  Met  requirements.  Data  used  for  verification  for  time  period, 
priority  deadlines  did  not  fully  test  the  time  period  due  to  the  values  used.  The 
following  are  considered  to  be  PARTIALLY  MET,  and  require  an  additional 
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verification  event  in  order  to  be  fully  assessed:  One  Test  Event  on  Test  Asset 
during  Time  Period,  Test  Event  Priority  Relationships,  High  to  Medium  Priority 
Relationship,  and  Test  Event  Priority  Deadlines. 

Not  Verified  requirements.  Data  did  not  include  low  priority  test  events, 
test  asset  type,  test  venue  capacity,  test  asset  unavailability,  test  venue 
unavailability,  and  medium  deadline.  The  following  are  considered  to  be  NOT 
VERIFIED,  and  require  an  additional  verification  event  in  order  to  be  assessed: 
Medium  to  Low  Priority  Relationship,  High  to  Low  Priority  Relationships,  Test 
Venue  Test  Asset  Capacity,  Test  Asset  Type  Test  Event  Relationships,  Test 
Asset  Unavailability,  Test  Venue  Unavailability,  Medium  Priority  to  Deadline 
Relationship,  and  Low  Priority  Schedule  Relationship. 
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D.  BETA  MODE  SCHEDULE 


Time  Period 

Test 

Asset 

P001 

P002 

P003 

P004 

P005 

P006 

P007 

P008 

P009 

POlO 

POll 

P012 

P013 

P014 

P015 

P016 

P017 

P018 

P019 

P020 

TA1 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

LM 

LM 

tZ5 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t06 

t06 

TA2 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t29 

t29 

TA3 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t36 

TA4 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

RGT1 

RGT1 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t25 

t56 

t56 

TA5 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

tl3 

tl3 

tl3 

tl3 

tl3 

tl3 

tl3 

tl3 

tl3 

tl3 

tl3 

tl3 

tl3 

tl3 

tl3 

tl3 

tl3 

tl3 

TA6 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

RGT1 

tl3 

tl3 

tl3 

tl3 

tl3 

tl3 

tl3 

tl3 

tl3 

tl3 

tl3 

tl3 

tl3 

tl3 

tl3 

tl3 

tl3 

tl3 

t58 

TA7 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

RGT1 

tl3 

tl3 

tl3 

tl3 

tl3 

tl3 

tl3 

tl3 

tl3 

tl3 

tl3 

tl3 

tl3 

tl3 

tl3 

tl3 

tl3 

tl3 

t57 

Figure  37.  First  Month  of  Model  Beta_mode  Detailed  Schedule 
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Time  Period 

Test 

Asset 

P021 

P022 

P023 

P024 

P025 

P026 

P027 

P028 

P029 

P030 

P031 

P032 

P033 

P034 

P035 

P036 

P037 

P038 

P039 

P040 

TA1 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

t06 

t06 

toi 

toi 

tl2 

tl2 

tl2 

tl2 

tl2 

tl2 

tl2 

tl2 

t05 

t05 

t05 

t05 

til 

til 

til 

til 

TA2 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

ATC 

ATC 

ATC 

ATC 

ATC 

ATC 

WM 

WM 

WM 

WM 

t29 

t29 

t29 

t29 

t29 

t29 

t29 

t29 

AVTB 

AVTB 

AVTB 

AVTB 

AVTB 

AVTB 

tl5 

tl5 

tl5 

tl5 

TA3 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

F 

t36 

t36 

t36 

t36 

t36 

t36 

t36 

t36 

t36 

t40 

t40 

t40 

t40 

t40 

t40 

t40 

t40 

t40 

t40 

t64 

TA4 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

t56 

t56 

t56 

t56 

t56 

t56 

t56 

t56 

t56 

t56 

t56 

t56 

t56 

t56 

t56 

t56 

t56 

t56 

t56 

t56 

AVTB 

AVTB 

AVTB 

TA5 

to 

to 

to 

S 

S 

S 

S 

S 

S 

S 

S 

S 

S 

S 

S 

S 

S 

S 

S 

S 

WSM 

R 

WSM 

R 

WSM 

R 

tS3 

t53 

t53 

tS3 

t53 

tS3 

t53 

t53 

t53 

t53 

t53 

t53 

t53 

t53 

t53 

t53 

t53 

TA6 

RGT1 

RGT1 

RGT1 

RGTX 

RGT1 

RGTX 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

t58 

t58 

t58 

t58 

t58 

t58 

t58 

t58 

t58 

t58 

t58 

t58 

t58 

t58 

t58 

t58 

t58 

t58 

t58 

t58 

TA7 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

t57 

t57 

t57 

t57 

t57 

t57 

t57 

t57 

t57 

t57 

t57 

t57 

t57 

t57 

t57 

t57 

t57 

t57 

t57 

t57 

Figure  38.  Second  Month  of  Model  Beta_mode  Detailed  Schedule 
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Time  Period 

Test 

Asset 

P041 

P042 

P043 

P044 

P045 

P046 

P047 

P048 

P049 

P050 

P051 

P052 

P053 

P0S4 

P055 

P056 

P057 

P0S8 

P059 

P060 

TA1 

LM 

LM 

LM 

LM 

S/HF 

S/HF 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

S/HF 

tio 

tio 

tio 

tio 

t27 

t27 

t02 

t02 

t02 

t02 

t08 

t08 

t08 

t08 

t08 

t08 

t39 

TA2 

S/HF 

S/HF 

S/HF 

S/HF 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

t33 

t33 

t33 

t33 

t59 

t59 

t59 

t59 

t59 

t59 

t59 

t59 

t59 

t59 

t59 

t59 

t59 

t59 

t59 

t59 

TA3 

F 

F 

F 

F 

F 

F 

F 

F 

F 

F 

F 

F 

F 

F 

F 

F 

F 

F 

F 

F 

t64 

t64 

t64 

t64 

t64 

t64 

t64 

t64 

t64 

t64 

t64 

t64 

t64 

t64 

t64 

t64 

t64 

t64 

t64 

t64 

TA4 

LM 

LM 

LM 

LM 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

LM 

LM 

LM 

LM 

LM 

LM 

t04 

t04 

t04 

t04 

t28 

t28 

t28 

t28 

t28 

t28 

t28 

t28 

t28 

t28 

t03 

t03 

t03 

t03 

t03 

t03 

TA5 

S 

S 

S 

S 

S 

S 

S 

S 

S 

S 

S 

S 

S 

WSM 

WSM 

WSM 

S/HF 

S/HF 

S/HF 

S/HF 

t53 

t53 

t53 

t53 

t53 

t53 

t53 

t53 

t53 

t53 

t53 

t53 

t53 

YPG 

YPG 

YPG 

t41 

t41 

t41 

t41 

TA6 

RGT1 

WM 

WM 

WM 

WM 

WM 

WM 

WM 

WM 

WM 

WM 

WM 

WM 

WM 

WM 

WM 

t58 

tl6 

tl6 

tl6 

tl6 

tl4 

tl4 

tl4 

tl4 

tl4 

tl4 

tl4 

tl4 

t!4 

tl4 

tl9 

TA7 

RGT1 

WM 

WM 

WM 

WM 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

WM 

WM 

WM 

t57 

t!7 

t!7 

t!7 

t!7 

t37 

t37 

t37 

t37 

t37 

t37 

t37 

t37 

t37 

t37 

t20 

t20 

t!9 

Figure  39.  Third  Month  of  Model  Betajmode  Detailed  Schedule 
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Time  Period 

Test 

Asset 

P061 

P062 

P063 

P064 

P065 

P066 

P067 

P068 

P069 

P070 

P071 

P072 

P073 

P074 

P075 

P076 

P077 

P078 

P079 

P080 

TA1 

S/HF 

t39 

S/HF 

t39 

S/HF 

t39 

S/HF 

t39 

S/HF 

t39 

S/HF 

t39 

S/HF 

t39 

S/HF 

t39 

S/HF 

t39 

S/HF 

t39 

S/HF 

t39 

S/HF 

t39 

S/HF 

t39 

S/HF 

t39 

S/HF 

t39 

S/HF 

t39 

S/HF 

t39 

S/HF 

t39 

S/HF 

t39 

S/HF 

t39 

TA2 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT2 

RGT2 

RGT2 

RGT2 

RGT2 

RGT2 

RGT2 

RGT2 

RGT2 

RGT2 

RGT2 

RGT2 

RGT2 

RGT2 

t59 

t59 

t59 

t59 

t59 

t59 

t61 

t61 

t61 

t61 

t61 

t61 

t61 

t61 

t61 

t61 

t61 

t61 

t61 

t61 

TA3 

F 

LM 

LM 

LM 

LM 

t64 

t07 

t07 

t07 

t07 

TA4 

LM 

LM 

LM 

LM 

LM 

LM 

t03 

t03 

t07 

t07 

t07 

t07 

TA5 

S/HF 

t41 

S/HF 

t41 

S/HF 

t41 

S/HF 

t41 

S/HF 

t41 

S/HF 

t41 

S/HF 

t41 

S/HF 

t41 

S/HF 

t41 

S/HF 

t41 

S/HF 

t41 

S/HF 

t41 

S/HF 

t41 

S/HF 

141 

S/HF 

t41 

S/HF 

t41 

S/HF 

t41 

S/HF 

t41 

S/HF 

t41 

S/HF 

t41 

TA6 

WM 

WM 

WM 

WM 

WM 

C 

C 

C 

C 

C 

C 

C 

C 

C 

C 

tl9 

tl8 

tl8 

tl8 

tl8 

t63 

t63 

t63 

t63 

t63 

t63 

t63 

t63 

t63 

t63 

TA7 

WM 

WM 

WM 

WM 

WM 

WM 

WM 

WM 

WM 

WM 

WM 

RGT2 

RGT2 

RGT2 

RGT2 

RGT2 

RGT2 

RGT2 

RGT2 

t!9 

t22 

t22 

t22 

t22 

t24 

t24 

t24 

t24 

t24 

t24 

t60 

t60 

t60 

t60 

t60 

t60 

t60 

t60 

Figure  40.  Fourth  Month  of  Model  Betajnode  Detailed  Schedule 
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Time  Period 

Test 

Asset 

P081 

P082 

P083 

P084 
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Figure  41.  Fifth  Month  of  Model  Beta_mode  Detailed  Schedule 
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Figure  42.  Sixth  Month  of  Model  Beta_mode  Detailed  Schedule 
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Table  50.  Model  Input  Files  to  Model  Beta_mode  Detailed  Schedule  Comparison 


Test 

Test 

Actual 

Test 

Actual 

Input 

Actual 

Successor 

Successor 

Priority 

Priority 

Event 

Asset 

Test 

Period 

Test 

Location 

Location 

Success? 

Issues?  i 

Input 

Assets 

Input 

Periods 

toi 

1 

1 

2 

2 

ATC  MD 

ATC 

t02 

yes 

high 

t02 

1 

1 

4 

4 

ATC  MD 

ATC 

high 

t03 

1 

1 

8 

8 

ATC  MD 

ATC 

high 

t04 

1 

1 

4 

4 

ATC  MD 

ATC 

high 

t05 

1 

1 

4 

4 

ATC  MD 

ATC 

high 

t06 

1 

1 

4 

4 

ATC  MD 

ATC 

high 

t07 

2 

2 

4 

4 

ATC  MD 

ATC 

medium 

t08 

1 

1 

6 

6 

ATC  MD 

ATC 

high 

t09 

1 

1 

20 

20 

ATC  MD 

ATC 

medium 

tio 

1 

1 

4 

4 

ATC  MD 

ATC 

high 

til 

1 

1 

4 

4 

ATC_MD 

ATC 

t09 

yes 

medium 

performed 
before 
high  tasks 

t12 

1 

1 

8 

8 

ATC  MD 

ATC 

high 

t13 

3 

3 

18 

18 

AVTBCA 

AVTB 

tl  4,  tl  5,  tl  6,  tl  7,  tl  8, 
tl  9,  t20,  t22,  t23,  t24, 
t63 

yes 

high 

t14 

1 

1 

10 

10 

AVTB  CA 

AVTB 

high 

t15 

1 

1 

4 

4 

AVTBCA 

AVTB 

tl  6,  tl  7,  tl  8 

yes 

high 

t16 

1 

1 

4 

4 

AVTB  CA 

AVTB 

high 

t17 

1 

1 

4 

4 

AVTB  CA 

AVTB 

high 

t18 

1 

1 

4 

4 

AVTB  CA 

AVTB 

high 

t19 

2 

1 

2 

2 

AVTB  CA 

AVTB 

high 

t20 

1 

1 

2 

2 

AVTB  CA 

AVTB 

high 

t22 

1 

1 

4 

4 

AVTB  CA 

AVTB 

high 

t23 

2 

2 

6 

6 

AVTB  CA 

AVTB 

medium 

t24 

1 

1 

6 

6 

AVTB  CA 

AVTB 

medium 
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Test 

Event 

Test 

Asset 

Input 

Actual 

Test 

Assets 

Test 

Period 

Input 

Actual 

Test 

Periods 

Input 

Location 

Actual 

Location 

t25 

4 

4 

18 

18 

ATC_MD 

ATC 

t27 

1 

1 

2 

2 

ATC  MD 

ATC 

t28 

1 

1 

10 

10 

ATC  MD 

ATC 

t29 

1 

1 

10 

10 

ATC  MD 

ATC 

t33 

1 

1 

4 

4 

AVTB  CA 

AVTB 

t36 

1 

1 

10 

10 

ATC  MD 

ATC 

t37 

1 

1 

10 

10 

AVTB  CA 

AVTB 

t39 

1 

1 

40 

40 

ATC  MD 

ATC 

t40 

1 

1 

10 

10 

ATC_MD 

ATC 

t41 

1 

1 

40 

40 

ATC  MD 

YPG  AZ 

YPG 

t53 

1 

1 

30 

30 

WSMRJMM 

WSMR 

t56 

1 

1 

22 

22 

ATC  MD 
AVTB  CA 
WSMR  NM 
YPG  AZ 

ATC 

t57 

1 

1 

22 

22 

ATC  MD 

AVTB 

AVTB_CA 
WSMRNM 
YPG  AZ 
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Successor 


Successor  Priority  Priority 
Success?  Issues? 


tOI ,  t02,  t03,  t04, 
t05,  t06,  t07,  t08, 
t09,  tIO,  til,  t12, 
tl  4,  tl  5,  tl  6,  tl  7, 
tl  8,  tl  9,  t20,  t22, 
t23,  t24,  t27,  t28, 
t29,  t33,  t36,  t39 
t40,  t41 ,  t53,  t64 

yes  high 

high 

high 

high 

high 

high 

high 

medium 

t39 

medium  performed 
before 
high  tasks 

medium 

t22 

medium  performed 
before 

high  tasks 

high 


high 


Test 

Event 

Test 

Asset 

Input 

Actual 

Test 

Assets 

Test 

Period 

Input 

Actual 

Test 

Periods 

Input 

Location 

Actual 

Location 

t58 

1 

1 

22 

22 

ATC  MD 
AVTB  CA 
WSMR  NM 
YPG  AZ 

AVTB 

t59 

1 

1 

22 

22 

ATC  MD 
AVTB  CA 
WSMR  NM 
YPG  AZ 

AVTB 

t60 

1 

1 

22 

22 

ATC  MD 
AVTB  CA 
WSMR  NM 
YPG_AZ 

AVTB 

t61 

1 

1 

22 

22 

ATC  MD 
AVTB  CA 
WSMR  NM 
YPG_AZ 

AVTB 

t62 

1 

1 

22 

22 

ATC  MD 
AVTB  CA 
WSMR  NM 
YPG  AZ 

ATC 

t63 

1 

1 

32 

32 

AVTB  CA 

AVTB 

t64 

1 

1 

22 

22 

ATC  MD 

ATC 
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Successor 


Successor  Priority  Priority 
Success?  Issues? 


high 


high 


medium 


medium 


medium 


medium 

high 


Using  the  information  provided  in  the  Beta_mode  schedule  and  the 
comparison  table,  the  verification  assessment  results  for  Beta_mode  schedule 
generated  by  the  model  is  given  in  the  Table  51  in  the  “Verification  &  Rationale” 
column,  which  is  relative  to  Table  4  requirements  that  are  repeated  here. 


Table  51 .  Model  Controls  and  Constraints  Beta_mode  Schedule 

Verification 


Para  # 

Para  Title 

Requirement 

Verification  Criteria 

Verification  & 
Rationale 

2.0 

Controls  and 
Constraints 

Not  Applicable 

Not  Applicable 

Not  Applicable 

2.1 

Test  Period 
(T=0) 

The  TE  RSCSP 
model  schedule 
shall  be 

constrained  by  the 
test  period. 

The  requirement  shall 
be  verified  by 
confirming  that  the 
schedule  is  within  the 
test  period  given  in  the 
test  period  input  file. 

MET 

Test  period  is  180 
and  model  test 
period  used  is  102 
days. 

2.1.1 

One  Test 

Event  on  Test 
Asset  during 
Time  Period 
(T=0) 

The  TE  RSCSP 
model  shall  not 
allow  more  than 
one  test  event 
during  a  time 
period  on  a  test 
asset. 

The  requirement  shall 
be  verified  by 
confirming  that  the 
schedule  does  not 
show  more  than  one 
test  event  in  a  time 
period  on  a  test  asset. 

PARTIALLY  MET 
Output  File  does  not 
assign  a  test  asset, 
but  in  aggregate 
does  not  exceed  the 
number  of  test 
assets. 

2.1.2 

Place  all  Test 
Events  in  Test 
Period  (T=0) 

The  TE  RSCSP 
model  shall  place 
all  identified  test 
events  in  a  test 
period. 

The  requirement  shall 
be  verified  by 
confirming  that 
schedule  places  all 
identified  test  events  in 
a  test  period. 

MET 

Actual  test  events 
from  input  file  are  all 
placed  in  test 
periods. 

2.1.3 

Test  Event 

Test  Period 

Durations 

(T=0) 

The  TE  RSCSP 
model  shall  use  the 
test  event  test 
period  durations 
input. 

The  requirement  shall 
be  verified  by 
confirming  that 
schedule  test  event  test 
period  durations 
matches  the  test  events 
test  period  input  file. 

MET 

Actual  test  event  test 
periods  tracks  to 
input  tile. 

2.2 

Test  Events 
(T=0) 

The  TE  RSCSP 
model  schedule 
shall  be 

constrained  by  the 
test  events. 

The  requirement  shall 
be  verified  by 
confirming  that  the 
schedule  includes  all 
test  events  included  in 
the  test  events  input 
file. 

MET 

Actual  test  events 
used  track  to  input 
file. 
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Para  # 

Para  Title 

Requirement 

Verification  Criteria 

Verification  & 
Rationale 

2.3 

Test  Asset 
Type  (0) 

The  TE  RSCSP 
model  shall  use 
test  asset  type 
inputs. 

The  requirement  shall 
be  verified  by 
confirming  that 
schedule  results  places 
test  events  on  the 
applicable  test  asset 
type  based  on  the  input 
file. 

NOT  VERIFIED 

Input  files  did  not 
use  more  than  one 
test  asset  type. 

2.4 

Test  Venues 
(T=0) 

The  TE  RSCSP 
model  shall  use 
test  venues  input. 

The  requirement  shall 
be  verified  by 
confirming  that  the 
schedule  includes  only 
the  test  venues 
included  in  the  test 
venues  input  file. 

MET 

Actual  test  venues 
used  track  to  input 
file. 

2.5 

Test  Event 
Priorities 

Not  Applicable 

Not  Applicable 

Not  Applicable 

2.5.1 

Test  Event 
Priority 
Relationships 
(T=0) 

The  TE  RSCSP 
model  shall  be 
constrained  by  the 
test  event  priority 
relationships 

The  requirement  shall 
be  verified  by 
confirming  that 
schedule  results  are 
constrained  by  the  test 
event  priority 
relationships  input  file. 

PARTIALLY  MET 

In  most  cases,  MET. 
There  is  an  issue 
with  til,  and  t40 
medium  test  events 
happening  before 
high  priority  tests. 

2.5.2 

High  to 

Medium 

Priority 

Relationship 

(T=0) 

The  TE  RSCSP 
model  high  priority 
test  events  shall  be 
started  before 
medium  priority  test 
events. 

The  requirement  shall 
be  verified  by 
confirming  that 
schedule  starts  high 
priority  test  events 
before  medium  priority 
test  events. 

PARTIALLY  MET 

In  most  cases,  MET. 
There  is  an  issue 
with  tl  1  and  t40 
medium  happening 
before  high  priority 
tests. 

2.5.3 

Medium  to 

Low  Priority 
Relationship 
(7=0) 

The  TE  RSCSP 
model  medium 
priority  test  events 
shall  be  started 
before  low  priority 
test  events. 

The  requirement  shall 
be  verified  by 
confirming  that 
schedule  starts  medium 
priority  test  events 
before  low  priority  test 
events. 

NOT  VERIFIED 

Model  run  did  not 
include  any  low 
priority  test  events. 

2.5.4 

High  to  Low 
Priority 
Relationships 
(7=0) 

The  TE  RSCSP 
model  high  priority 
test  events  shall  be 
started  before  low 
priority  test  events. 

The  requirement  shall 
be  verified  by 
confirming  that 
schedule  starts  high 
priority  test  events 
before  low  priority  test 
events. 

NOT  VERIFIED 

Model  run  did  not 
include  any  low 
priority  test  events. 

2.6 

Test  Venue 
Movement 

Test  Periods 
(T=0) 

The  TE  RSCSP 
model  shall  be 
constrained  by  the 
test  venue 
movement  test 
periods. 

The  requirement  shall 
be  verified  by 
confirming  that 
schedule  uses  the  test 
venue  movement  test 
periods  based  on  the 
input  file. 

MET 

Data  shows  that 
when  moving 
venues,  the  input 
data  matches  the 
input  file. 
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Para  # 

Para  Title 

Requirement 

Verification  Criteria 

Verification  & 
Rationale 

2.6.1 

Add  Time 
Period  from 
Test  Venue 
Movement 
(T=0) 

The  TE  RSCSP 
model  shall  add 
time  periods  to  the 
schedule  based  on 
the  distance 
between  test 

venues. 

The  requirement  shall 
be  verified  by 
confirming  that 
schedule  results  show 
that  the  time  periods 
added  to  the  schedule 
when  the  test  asset 
moves  to  a  new  test 
venue  are  based  on  the 
input  file. 

MET 

Data  shows  that 
when  moving 
venues,  the  input  file 
test  periods  are 
added  to  the  test 
schedule. 

2.6.2 

Schedule  Test 
Event  on 
Available  Test 
Venues  (T=0) 

The  TE  RSCSP 
model  shall  not 
schedule  a  test 
event  at  a  test 
venue  that  does  not 
perform  that  test 
event. 

The  requirement  shall 
be  verified  by 
confirming  that 
schedule  results  show 
that  the  test  events  are 
scheduled  only  on 
available  test  venues 
based  on  the  input  file. 

MET 

Test  venues  used 
track  to  test  venues 
allowed  based  on 
the  input  file. 

2.7 

Test  Venue 
Test  Asset 
Capacity 
(T=0) 

The  TE  RSCSP 
model  shall  be 
constrained  by  test 
venue  test  asset 
capacity  input. 

The  requirement  shall 
be  verified  by 
confirming  that  the 
number  of  test  assets 
located  at  each  test 
venue  on  the  schedule 
does  not  exceed  the 
capacity  identified  in 
the  input  file. 

NOT  VERIFIED 

The  number  of  test 
assets  assigned  to  a 
test  venue  by  the 
model  did  not 
exceed  the  amount 
in  the  file,  but  the 
numbers  in  the  file 
(10)  did  not  really 
check  the  model. 

2.8 

Test  Asset 

Test  Venue 
Starting 
Location 
(T=0) 

The  TE  RSCSP 
model  shall  be 
constrained  by  the 
test  asset  test 
venue  starting 
location  input. 

The  requirement  shall 
be  verified  by 
confirming  that 
schedule  test  assets 
start  at  the  venues 
identified  in  the  test 
asset  test  venue  input 
file. 

MET 

The  starting 
locations  and 
quantities  (4  at  ATC 
and  3  at  AVTB)  track 
to  the  input  file. 

2.9 

Test  Event 

Test  Venue 

Relationships 

(T=0) 

The  TE  RSCSP 
model  shall  be 
constrained  by  the 
test  event  test 
venue  relationships 
input. 

The  requirement  shall 
be  verified  by 
confirming  that 
schedule  test  events 
are  performed  at  the 
test  venues  identified  in 
the  test  event  test 
venue  relationships 
input  file. 

MET 

Test  events  are 
assigned  to  allowed 
test  venues  based 
on  the  input  file. 

2.10 

Test  Event 
Precedence 
Relationships 
(T=0) 

The  TE  RSCSP 
model  shall  be 
constrained  by  test 
event  precedence 
relationships  input. 

The  requirement  shall 
be  verified  by 
confirming  that 
schedule  results  are 
constrained  by  the  test 
event  precedence 
relationships  input  file. 

MET 

Data  shows  that 
predecessor  test 
events  in  the 
schedule  occur 
before  successor 
test  events. 
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Para  # 

Para  Title 

Requirement 

Verification  Criteria 

Verification  & 
Rationale 

2.10.1 

Test  Event 
Predecessor 
First  (T=0) 

The  TE  RSCSP 
model  shall  perform 
predecessor  test 
events  prior  to  test 
event  successor 
test  events. 

The  requirement  shall 
be  verified  by 
confirming  that 
predecessor  test 
events  are  on  the 
schedule  before 
successor  test  events 
regardless  of  test  event 
priority. 

MET 

Data  shows  that 
predecessor  test 
events  in  the 
schedule  occur 
before  successor 
test  events. 

2.10.2 

Test  Event 
Successor 
Second  (T=0) 

The  TE  RSCSP 
model  shall  not 
perform  successor 
test  events  prior  to 
test  event 
predecessor  test 
events. 

The  requirement  shall 
be  verified  by 
confirming  that 
successor  test  events 
are  on  the  schedule 
before  predecessor  test 
events  regardless  of 
test  event  priority. 

MET 

Data  shows  that 
successor  test 
events  in  the 
schedule  occur  after 
predecessor  test 
events. 

2.11 

Number  of 

Test  Assets 
Needed  for 

Test  Events 
(T=0) 

The  TE  RSCSP 
model  shall  use  the 
number  of  test 
assets  needed  for 
test  events  input. 

The  requirement  shall 
be  verified  by 
confirming  that  the 
schedule  uses  the 
number  of  test  assets 
needed  for  test  events 
input  file. 

MET 

Data  shows  that  the 
number  of  test 
assets  needed  for  a 
test  event  are  used. 
For  multi-vehicle 
operations,  tests  are 
performed 
simultaneously  on 
the  test  assets. 

2.12 

Low  Priority 
Schedule 
Relationship 
(0) 

The  TE  RSCSP 
model  low  priority 
test  events  shall  be 
allowed  to  go 
beyond  the  test 
period. 

The  requirement  shall 
be  verified  by 
confirming  that 
schedule  results  match 
the  input  file. 

NOT  VERIFIED 

Low  priority  test 
events  are  not  used 
in  this  model  run. 

2.13 

Test  Asset 
Unavailability 
(0) 

The  TE  RSCSP 
model  shall  be 
constrained  by  the 
test  asset 
unavailability. 

The  requirement  shall 
be  verified  by 
confirming  that 
schedule  places  test 
assets  only  on 
available  test  assets 
based  on  the  input  file. 

NOT  VERIFIED 

Data  for  Test  Asset 
Unavailability  is  not 
included  in  the 
model  run. 

2.14 

Test  Venue 
Unavailability 
(0) 

The  TE  RSCSP 
model  shall  be 
constrained  by  test 
venue 

unavailability. 

The  requirement  shall 
be  verified  by 
confirming  that 
schedule  results  are 
constrained  by  the 
input  file. 

NOT  VERIFIED 

Data  for  Test  Venue 
Unavailability  is  not 
included  in  the 
model  run. 

2.15 

Test  Event 
Priority 
Deadlines 
(T=0) 

The  TE  RSCSP 
model  shall  be 
constrained  by  the 
high  and  medium 
test  event  priority 
deadlines. 

The  requirement  shall 
be  verified  by 
confirming  that 
schedule  results  are 
constrained  by  the 
input  file. 

PARTIALLY  MET 

This  model  run  used 

1 90  for  the  total  test 
periods.  High  is  170, 
and  medium  is  190. 

175 


Para  # 

Para  Title 

Requirement 

Verification  Criteria 

Verification  & 
Rationale 

2.15.1 

High  Priority  to 
Deadline 
Relationship 
(T-O) 

TheTE  RSCSP 
model  shall 
complete  high 
priority  test  events 
before  high  priority 
deadlines. 

The  requirement  shall 
be  verified  by 
confirming  that 
schedule  results  start 
high  priority  test  events 
before  the  high  priority 
deadline. 

MET 

2.15.2 

Medium 

Priority  to 
Deadline 
Relationship 
(7=0) 

The  TE  RSCSP 
model  shall 
complete  medium 
priority  test  events 
before  medium 
priority  deadlines. 

The  requirement  shall 
be  verified  by 
confirming  that 
schedule  results  start 
medium  priority  test 
events  before  the 
medium  priority 
deadline. 

NOT  VERIFIED 

This  model  run  used 

1 90  for  the  total  test 
periods  and  the 
medium  test 
periods,  which 
means  that  it  is  not 
verified. 

2.15.3 

High  Priority  to 
Test  Period 
Relationship 
(7=0) 

TheTE  RSCSP 
model  shall 
complete  high 
priority  test  events 
before  the  test 
period  completes. 

The  requirement  shall 
be  verified  by 
confirming  that 
schedule  results 
complete  high  priority 
test  events  before  the 
test  period  completes. 

MET 

High  priority  test 
events  end  at  66. 
Schedule  ends  at 
test  period  102.  Test 
period  ends  at  190. 

2.15.4 

Medium 

Priority  to  Test 
Period 
Relationship 
(T=0) 

The  TE  RSCSP 
model  shall 
complete  medium 
priority  test  events 
before  the  test 
period  completes. 

The  requirement  shall 
be  verified  by 
confirming  that 
schedule  results 
complete  medium 
priority  test  events 
before  the  test  period 
completes. 

MET 

Medium  priority  test 
events  end  at  test 
period  102. 

Schedule  ends  at 
test  period  102 

Test  period  ends  at 
190. 
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The  results  of  the  model  controls  and  constraints  requirements  schedule 
verification  assessment  for  Betajmode  indicate  that  the  majority  of  the  threshold 
model  controls  and  constraints  requirements  in  Table  51  are  MET,  except  for 
some  that  are  PARTIALLY  MET  and  some  that  are  NOT  VERIFIED.  There  are 
no  controls  and  constraints  requirements  that  are  verified  to  be  NOT  MET. 

Partially  Met  requirements.  Data  used  for  verification  for  time  period, 
priority  deadlines  did  not  fully  test  the  time  period  due  to  the  values  used.  The 
following  are  considered  to  be  PARTIALLY  MET,  and  require  an  additional 
verification  event  in  order  to  be  fully  assessed:  One  Test  Event  on  Test  Asset 
during  Time  Period,  Test  Event  Priority  Relationships,  High  to  Medium  Priority 
Relationship,  and  Test  Event  Priority  Deadlines. 

Not  Verified  requirements.  Data  did  not  include  low  priority  test  events, 
test  asset  type,  test  venue  capacity,  test  asset  unavailability,  test  venue 
unavailability,  and  medium  deadline.  The  following  are  considered  to  be  NOT 
VERIFIED,  and  require  an  additional  verification  event  in  order  to  be  assessed: 
Medium  to  Low  Priority  Relationship,  High  to  Low  Priority  Relationships,  Test 
Venue  Test  Asset  Capacity,  Test  Asset  Type  Test  Event  Relationships,  Test 
Asset  Unavailability,  Test  Venue  Unavailability,  Medium  Priority  to  Deadline 
Relationship,  and  Low  Priority  Schedule  Relationship. 
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E. 


BETA  MEAN  SCHEDULE 


Time  Period 

Test 

Asset 

P001 

P002 

P003 

P004 

P005 

P006 

P007 

P008 

P009 

P010 

POll 

P012 

P013 

P014 

P015 

P016 

P017 

P018 

P019 

P020 

TA1 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

TA2 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

TA3 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

TA4 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

S/HF 

t25 

TA5 

S/HF 

tl3 

S/HF 

tl3 

S/HF 

tl3 

S/HF 

tl3 

S/HF 

tl3 

S/HF 

tl3 

S/HF 

tl3 

S/HF 

tl3 

S/HF 

tl3 

S/HF 

tl3 

S/HF 

tl3 

S/HF 

tl3 

S/HF 

tl3 

S/HF 

tl3 

S/HF 

tl3 

S/HF 

tl3 

S/HF 

tl3 

S/HF 

tl3 

S/HF 

tl3 

TA6 

S/HF 

tl3 

S/HF 

tl3 

S/HF 

tl3 

S/HF 

tl3 

S/HF 

tl3 

S/HF 

tl3 

S/HF 

tl3 

S/HF 

tl3 

S/HF 

tl3 

S/HF 

tl3 

S/HF 

tl3 

S/HF 

tl3 

S/HF 

tl3 

S/HF 

tl3 

S/HF 

tl3 

S/HF 

tl3 

S/HF 

tl3 

S/HF 

tl3 

S/HF 

tl3 

TA7 

S/HF 

tl3 

S/HF 

tl3 

S/HF 

tl3 

S/HF 

tl3 

S/HF 

t!3 

S/HF 

tl3 

S/HF 

tl3 

S/HF 

tl3 

S/HF 

tl3 

S/HF 

tl3 

S/HF 

tl3 

S/HF 

tl3 

S/HF 

tl3 

S/HF 

tl3 

S/HF 

tl3 

S/HF 

tl3 

S/HF 

tl3 

S/HF 

tl3 

S/HF 

tl3 

Figure  43.  First  Month  of  Model  Betajmean  Detailed  Schedule 
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Time  Period 

Test 

Asset 

P021 

P022 

P023 

P024 

P025 

P026 

P027 

P028 

P029 

P030 

P031 

P032 

P033 

P034 

P035 

P036 

P037 

P038 

P039 

P040 

TA1 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

t08 

t08 

t08 

t08 

t08 

t08 

t08 

tl2 

tl2 

tl2 

tl2 

tl2 

tl2 

tl2 

tl2 

tl2 

t03 

t03 

t03 

TA2 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

F 

F 

F 

F 

F 

F 

F 

t27 

t27 

t40 

t40 

t40 

t40 

t40 

t40 

t40 

t40 

t40 

t40 

t40 

t64 

t64 

t64 

t64 

t64 

t64 

t64 

TA3 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

t57 

t57 

t57 

t57 

t57 

t57 

t57 

t57 

t57 

t57 

t57 

t57 

t57 

t57 

t57 

t57 

t57 

t57 

t57 

t57 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

TA4 

t59 

t59 

tS9 

t59 

t59 

t59 

t59 

t59 

t59 

t59 

t59 

t59 

t59 

t59 

t59 

t59 

t59 

t59 

tS9 

TA5 

S/HF 

AVTBto 

AVTBto 

AVTBto 

S 

S 

S 

S 

S 

S 

S 

S 

S 

S 

S 

S 

S 

S 

S 

tl3 

WSMR 

WSMR 

WSMR 

t53 

t53 

t53 

t53 

t53 

t53 

t53 

t53 

t53 

t53 

t53 

t53 

t53 

t53 

t53 

TA6 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

WM 

WM 

WM 

WM 

WM 

WM 

WM 

WM 

tl3 

t33 

t33 

t33 

t33 

t33 

tl5 

tlS 

tl5 

tl5 

tl5 

t20 

t20 

tl8 

TA7 

S/HF 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

t!3 

t56 

t56 

t56 

t56 

tS6 

t56 

t56 

t56 

t56 

t56 

t56 

t56 

t56 

t56 

t56 

t56 

t56 

t56 

t56 

Figure  44.  Second  Month  of  Model  Beta_mean  Detailed  Schedule 
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Time  Period 

Test 

Asset 

P041 

P042 

P043 

P044 

P045 

P046 

P047 

P048 

P049 

P050 

P051 

P052 

P053 

P054 

P055 

P056 

P057 

P058 

P059 

P060 

TA1 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

t03 

t03 

t03 

t03 

t03 

t03 

toi 

toi 

tio 

tio 

tio 

tio 

tio 

t36 

t36 

t36 

t36 

t36 

t36 

t36 

TA2 

F 

F 

F 

F 

F 

F 

F 

F 

F 

F 

F 

F 

F 

F 

F 

F 

S/HF 

S/HF 

S/HF 

t64 

t64 

t64 

t64 

t64 

t64 

t64 

t64 

t64 

t64 

t64 

t64 

t64 

t64 

t64 

t64 

t28 

t28 

t28 

TA3 

RGT1 

RGT1 

LM 

LM 

LM 

LM 

LM 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

RGT1 

t57 

t57 

toe 

t06 

t06 

t06 

t06 

t58 

t58 

t58 

t58 

t58 

tS8 

t58 

t58 

t58 

t58 

t58 

t58 

t58 

TA4 

RGT1 

RGT1 

RGT1 

LM 

LM 

LM 

LM 

LM 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

t59 

t59 

t59 

til 

til 

til 

til 

til 

t29 

t29 

t29 

t29 

t29 

t29 

t29 

t29 

t29 

t29 

t29 

t05 

TA5 

S 

S 

S 

S 

S 

S 

S 

S 

S 

S 

S 

S 

S 

S 

S 

S 

S 

WSMR 

WSMR 

WSMR 

t53 

t53 

t53 

t53 

t53 

t53 

t53 

t53 

t53 

t53 

t53 

t53 

t53 

t53 

t53 

t53 

t53 

toYPG 

toYPG 

toYPG 

TA6 

WM 

WM 

WM 

WM 

WM 

WM 

WM 

WM 

WM 

WM 

WM 

WM 

WM 

WM 

WM 

WM 

tis 

tl8 

tl8 

tl8 

tl4 

tl4 

tl4 

tl4 

tl4 

tl4 

tl4 

tl4 

tl4 

tl4 

tl4 

tie 

TA7 

RGT1 

RGT1 

RGT1 

WM 

WM 

WM 

WM 

WM 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

t56 

t56 

t56 

t!7 

t!7 

t!7 

t!7 

t!7 

t37 

t37 

t37 

t37 

t37 

t37 

t37 

t37 

t37 

t37 

t37 

Figure  45.  Third  Month  of  Model  Beta_mean  Detailed  Schedule 
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Time  Period 

Test 

Asset 

P061 

P062 

P063 

P064 

P065 

P066 

P067 

P068 

P069 

P070 

P071 

P072 

P073 

P074 

P075 

P076 

P077 

P078 

P079 

P080 

TA1 

S/HF 

S/HF 

S/HF 

S/HF 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

LM 

t36 

t36 

t36 

t36 

t02 

t02 

t02 

t02 

t02 

t09 

t09 

t09 

t09 

t09 

t09 

t09 

t09 

t09 

t09 

t09 

TA2 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

t28 

t28 

t28 

t28 

t28 

t28 

t28 

t28 

t39 

t39 

t39 

t39 

t39 

t39 

t39 

t39 

t39 

t39 

t39 

t39 

RGT 

RGT 

RGT 

RGT 

RGT 

RGT 

RGT 

RGT 

RGT 

TA3 

RGT2 

RGT2 

RGT2 

RGT2 

RGT2 

RGT2 

RGT2 

RGT2 

RGT2 

RGT2 

RGT2 

t61 

t61 

t61 

t61 

t61 

t61 

t61 

t61 

t61 

t61 

t61 

t58 

t58 

t58 

t58 

t58 

t58 

t58 

t58 

t58 

TA4 

S/HF 

S/HF 

S/HF 

S/HF 

LM 

LM 

LM 

LM 

LM 

RGT2 

RGT2 

RGT2 

RGT2 

t05 

t05 

t05 

t05 

t04 

t04 

t04 

t04 

t04 

t62 

t62 

t62 

t62 

TA5 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

S/HF 

t41 

t41 

t41 

t41 

t41 

t41 

t41 

t41 

t41 

t41 

t41 

t41 

t41 

t41 

t41 

t41 

t41 

t41 

t41 

t41 

TA6 

WM 

WM 

WM 

WM 

WM 

WM 

WM 

WM 

WM 

WM 

WM 

WM 

WM 

WM 

WM 

WM 

WM 

WM 

t!6 

tl6 

tl6 

tl6 

t23 

t23 

t23 

t23 

t23 

t23 

t23 

t23 

t24 

t24 

t24 

t24 

t24 

t24 

TA7 

WM 

WM 

WM 

WM 

WM 

WM 

WM 

WM 

WM 

WM 

WM 

WM 

WM 

C 

C 

C 

C 

C 

C 

t22 

t22 

t22 

t22 

t22 

t23 

t23 

t23 

t23 

t23 

t23 

t23 

t23 

t63 

t63 

t63 

t63 

t63 

t63 

Figure  46.  Fourth  Month  of  Model  Betajnean  Detailed  Schedule 
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Time  Period 

Test 

Asset 

P081 

P082 

P083 

P084 

P085 

P086 

P087 

P088 

P089 

P090 

P091 

P092 

P093 

P094 

P095 

P096 

P097 

P098 

P099 

P100 

TA1 

LM 

t09 

LM 

t09 

LM 

t09 

LM 

t09 

LM 

t09 

LM 

t09 

LM 

t09 

LM 

t09 

LM 

t09 

LM 

t09 

LM 

t09 

LM 

t07 

LM 

t07 

LM 

t07 

LM 

t07 

LM 

t07 

TA2 

S/HF 

t39 

S/HF 

t39 

S/HF 

t39 

S/HF 

t39 

S/HF 

t39 

S/HF 

t39 

S/HF 

t39 

S/HF 

t39 

S/HF 

t39 

S/HF 

t39 

S/HF 

t39 

S/HF 

t39 

S/HF 

t39 

S/HF 

t39 

S/HF 

t39 

S/HF 

t39 

S/HF 

t39 

S/HF 

t39 

S/HF 

t39 

S/HF 

t39 

TA3 

RGT2 

t61 

RGT2 

t61 

RGT2 

t61 

RGT2 

t61 

RGT2 

t61 

RGT2 

t61 

RGT2 

t61 

RGT2 

t61 

RGT2 

t61 

RGT2 

t61 

RGT2 

t61 

LM 

t07 

LM 

t07 

LM 

t07 

LM 

t07 

LM 

t07 

TA4 

RGT2 

t62 

RGT2 

t62 

RGT2 

t62 

RGT2 

t62 

RGT2 

t62 

RGT2 

t62 

RGT2 

t62 

RGT2 

t62 

RGT2 

t62 

RGT2 

t62 

RGT2 

t62 

RGT2 

t62 

RGT2 

t62 

RGT2 

t62 

RGT2 

t62 

RGT2 

t62 

RGT2 

t62 

RGT2 

t62 

TA5 

S/HF 

t41 

S/HF 

t41 

S/HF 

t41 

S/HF 

t41 

S/HF 

t41 

S/HF 

t41 

S/HF 

t41 

S/HF 

t41 

S/HF 

t41 

S/HF 

t41 

S/HF 

t41 

S/HF 

t41 

S/HF 

t41 

S/HF 

t41 

S/HF 

t41 

S/HF 

t41 

S/HF 

t41 

S/HF 

t41 

S/HF 

t41 

S/HF 

t41 

TA6 

WM 

t24 

RGT2 

t60 

RGT2 

t60 

RGT2 

t60 

RGT2 

t60 

RGT2 

t60 

RGT2 

t60 

RGT2 

L60 

RGT2 

t60 

RGT2 

t60 

RGT2 

t60 

RGT2 

t60 

RGT2 

t60 

RGT2 

t60 

RGT2 

t60 

RGT 

2 

t60 

RGT 

2 

t60 

TA7 

C 

t63 

C 

t63 

C 

t63 

C 

t63 

C 

t63 

C 

t63 

C 

t63 

C 

t63 

C 

t63 

C 

t63 

C 

t63 

C 

t63 

C 

t63 

C 

t63 

C 

t63 

C 

t63 

C 

t63 

C 

t63 

C 

t63 

C 

t63 

Figure  47.  Fifth  Month  of  Model  Beta_mean  Detailed  Schedule 
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Time  Period 

Test 

Asset 

P101 

P102 

P103 

P104 

P105 

P106 

P107 

P108 

P109 

P110 

Pill 

P112 

P113 

P114 

P115 

P116 

P117 

P118 

P119 

P120 

TA1 

TA2 

S/HF 

t39 

S/HF 

t39 

S/HF 

t39 

S/HF 

t39 

S/HF 

t39 

S/HF 

t39 

S/HF 

t39 

S/HF 

t39 

S/HF 

t39 

S/HF 

t39 

S/HF 

t39 

TA3 

TA4 

TA5 

S/HF 

t41 

S/HF 

t41 

S/HF 

t41 

TA6 

RGT2 

t60 

RGT2 

t60 

RGT2 

t60 

RGT2 

t60 

RGT2 

t60 

RGT2 

1 60 

WM 

tl9 

WM 

tl9 

TA7 

C 

t63 

C 

t63 

C 

t63 

C 

t63 

C 

t63 

C 

t63 

C 

t63 

WM 

t!9 

WM 

t!9 

Figure  48.  Sixth  Month  of  Model  Betajmean  Detailed  Schedule 
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1 

1 

1 

1 

1 

1 

2 

1 

1 

1 

1 

1 

3 

1 

1 

1 

1 

1 

2 

1 

1 

2 

1 


Table  52.  Model  Input  Files  to  Model  Beta_mean  Detailed  Schedule  Comparison 


Actual 

Test 

Assets 

Test 
Period 
Av  Calc 

Actual 

Beta 

Test 

Periods 

Input 

Location 

Actual 

Location 

Successor 

Successor 

Success? 

Priority 

Priority 

Issues? 

1 

2 

2 

ATC  MD 

ATC 

t02 

yes 

high 

1 

4 

5 

ATC  MD 

ATC 

high 

1 

8 

9 

ATC  MD 

ATC 

high 

1 

4 

5 

ATC  MD 

ATC 

high 

1 

4 

5 

ATC  MD 

ATC 

high 

1 

4 

5 

ATC  MD 

ATC 

high 

2 

4 

5 

ATC  MD 

ATC 

medium 

1 

6 

7 

ATC  MD 

ATC 

high 

1 

20 

22 

ATC  MD 

ATC 

medium 

1 

4 

5 

ATC  MD 

ATC 

high 

1 

4 

5 

AT  C_M  D 

ATC 

t09 

yes 

medium 

performed 
before 
high  tasks 

1 

8 

9 

ATC  MD 

ATC 

high 

3 

18 

20 

AVTBCA 

AVTB 

tl  4,  tl  5,  tl  6,  tl  7, 
tl  8,  tl  9,  t20,  t22, 

yes 

high 

t23,  t24,  t63 

1 

10 

11 

AVTB  CA 

AVTB 

high 

1 

4 

5 

AVTB_CA 

AVTB 

tl  6,  tl  7,  tl  8 

yes 

high 

1 

4 

5 

AVTB  CA 

AVTB 

high 

1 

4 

5 

AVTB  CA 

AVTB 

high 

1 

4 

5 

AVTB  CA 

AVTB 

high 

2 

2 

2 

AVTBCA 

AVTB 

high 

Performe 
d  at  end 
after  low 

priority 

tasks 

1 

2 

2 

AVTB  CA 

AVTB 

high 

1 

4 

5 

AVTB  CA 

AVTB 

high 

2 

6 

8 

AVTB  CA 

AVTB 

medium 

1 

6 

7 

AVTB  CA 

AVTB 

medium 
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Test 

Event 

Test 

Asset 

Input 

Actual 

Test 

Assets 

Test 
Period 
Av  Calc 

Actual 

Beta 

Test 

Periods 

Input 

Location 

t25 

4 

4 

18 

20 

ATC_MD 

t27 

1 

1 

2 

2 

ATC  MD 

t28 

1 

1 

10 

11 

ATC  MD 

t29 

1 

1 

10 

11 

ATC  MD 

t33 

1 

1 

4 

5 

AVTB  CA 

t36 

1 

1 

10 

11 

ATC  MD 

t37 

1 

1 

10 

11 

AVTB  CA 

t39 

1 

1 

40 

43 

ATC  MD 

t40 

1 

1 

10 

10 

ATC_MD 

t41 

1 

1 

40 

43 

ATC  MD 
YPG_AZ 

t53 

1 

1 

30 

32 

WSMRJMM 

t56 

1 

1 

22 

22 

ATC  MD 
AVTB  CA 
WSMR  NM 
YPG  AZ 

t57 

1 

1 

22 

22 

ATC  MD 

Actual 

Location 


Successor 


Successor 

Success? 


Priority 


Priority 

Issues? 


ATC  tOI,  t02,  t03,  t04,  yes  high 

t05,  t06,  t07,  t08, 
t09,  tIO,  til,  tl 2, 
tl  4,  tl  5,  tl  6,  tl  7, 
tl  8,  tl  9,  t20,  t22, 
t23,  t24,  t27,  t28, 
t29,  t33,  t36,  t39 
t40,  t41 ,  t53,  t64 


ATC 

high 

ATC 

high 

ATC 

high 

AVTB 

high 

ATC 

high 

AVTB 

high 

ATC 

medium 

ATC 

t39 

yes 

medium 

performed 
before 
high  tasks 

YPG 

medium 

WSMR 

t22 

yes 

medium 

performed 
before 
high  tasks 

AVTB 

high 

t57 


22  22 
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Test 

Event 

Test 

Asset 

Input 

Actual 

Test 

Assets 

Test 
Period 
Av  Calc 

Actual 

Beta 

Test 

Periods 

Input 

Location 

t58 

1 

1 

22 

22 

ATC  MD 
AVTB  CA 
WSMR  NM 
YPG  AZ 

t59 

1 

1 

22 

22 

ATC  MD 
AVTB  CA 
WSMR  NM 
YPG  AZ 

t60 

1 

1 

22 

22 

ATC  MD 
AVTB  CA 
WSMR  NM 
YPG  AZ 

t61 

1 

1 

22 

22 

ATC  MD 
AVTB  CA 
WSMR  NM 
YPG  AZ 

t62 

1 

1 

22 

22 

ATC  MD 
AVTB  CA 
WSMR  NM 
YPG  AZ 

t63 

1 

1 

32 

33 

AVTB  CA 

t64 

1 

1 

22 

23 

ATC  MD 

Actual 

Location 


Successor 


Successor 

Success? 


Priority 


Priority 

Issues? 


ATC 

AVTB 

ATC 

ATC 

AVTB 

ATC 


high 


high 


medium 


medium 


medium 


medium 

high 
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Using  the  information  provided  in  the  Beta_mean  schedule  and  the 
comparison  table,  the  verification  assessment  results  for  Betajmean  schedule 
generated  by  the  model  is  given  in  the  Table  53  in  the  “Verification  &  Rationale” 
column,  which  is  relative  to  Table  4  requirements  that  are  repeated  here 


Table  53.  Model  Controls  and  Constraints  Betajmean  Schedule 

Verification 


Para  # 

Para  Title 

Requirement 

Verification  Criteria 

Verification  & 
Rationale 

2.0 

Controls  and 
Constraints 

Not  Applicable 

Not  Applicable 

Not  Applicable 

2.1 

Test  Period 
(T=0) 

The  TE  RSCSP 
model  schedule 
shall  be 
constrained  by 
the  test  period. 

The  requirement  shall  be 
verified  by  confirming  that 
the  schedule  is  within  the 
test  period  given  in  the  test 
period  input  file. 

MET 

Test  period  is 

180  and  model 
test  period  used 
is  1 1 1  days. 

2.1.1 

One  Test  Event 
on  Test  Asset 
during  Time 
Period  (T=0) 

The  TE  RSCSP 
model  shall  not 
allow  more  than 
one  test  event 
during  a  time 
period  on  a  test 
asset. 

The  requirement  shall  be 
verified  by  confirming  that 
the  schedule  does  not  show 
more  than  one  test  event  in 
a  time  period  on  a  test 
asset. 

PARTIALLY 

MET 

Output  File  does 
not  assign  a  test 
asset,  but  in 
aggregate  does 
not  exceed  the 
number  of  test 
assets. 

2.1.2 

Place  all  Test 
Events  in  Test 
Period  (T=0) 

The  TE  RSCSP 
model  shall  place 
all  identified  test 
events  in  a  test 
period. 

The  requirement  shall  be 
verified  by  confirming  that 
schedule  places  all 
identified  test  events  in  a 
test  period. 

MET 

Actual  test 
events  from  input 
file  are  all  placed 
in  test  periods. 

2.1.3 

Test  Event  Test 
Period 

Durations  (T=0) 

The  TE  RSCSP 
model  shall  use 
the  test  event  test 
period  durations 
input. 

The  requirement  shall  be 
verified  by  confirming  that 
schedule  test  event  test 
period  durations  matches 
the  test  events  test  period 
input  file. 

MET 

Actual  test  event 
test  periods 
tracks  to  input 
tile. 

2.2 

Test  Events 
(T=0) 

The  TE  RSCSP 
model  schedule 
shall  be 
constrained  by 
the  test  events. 

The  requirement  shall  be 
verified  by  confirming  that 
the  schedule  includes  all 
test  events  included  in  the 
test  events  input  file. 

MET 

Actual  test 
events  used 
track  to  input  file. 

2.3 

Test  Asset  Type 
(0) 

The  TE  RSCSP 
model  shall  use 
test  asset  type 
inputs. 

The  requirement  shall  be 
verified  by  confirming  that 
schedule  results  places  test 
events  on  the  applicable 
test  asset  type  based  on  the 
input  file. 

NOT  VERIFIED 
Input  files  did  not 
use  more  than 
one  test  asset 
type. 
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Para  #  Para  Title  Requirement  Verification  Criteria  Verification  & 


2.4 

Test  Venues 
(T=0) 

The  TE  RSCSP 
model  shall  use 
test  venues  input. 

The  requirement  shall  be 
verified  by  confirming  that 
the  schedule  includes  only 
the  test  venues  included  in 
the  test  venues  input  file. 

MET 

Actual  test 
venues  used 
track  to  input  file. 

2.5 

Test  Event 
Priorities 

Not  Applicable 

Not  Applicable 

Not  Applicable 

2.5.1 

Test  Event 
Priority 
Relationships 
(T=0) 

The  TE  RSCSP 
model  shall  be 
constrained  by 
the  test  event 
priority 
relationships 

The  requirement  shall  be 
verified  by  confirming  that 
schedule  results  are 
constrained  by  the  test 
event  priority  relationships 
input  file. 

PARTIALLY 

MET 

In  most  cases, 
MET.  There  is  an 
issue  with  tl  1, 
t40,  and  t53 
medium  test 
events 
happening 
before  high 
priority  tests. 

2.5.2 

High  to  Medium 
Priority 
Relationship 
(T=0) 

The  TE  RSCSP 
model  high 
priority  test  events 
shall  be  started 
before  medium 
priority  test 
events. 

The  requirement  shall  be 
verified  by  confirming  that 
schedule  starts  high  priority 
test  events  before  medium 
priority  test  events. 

PARTIALLY 

MET 

In  most  cases, 
MET.  There  is  an 
issue  with  tl  1, 
t40,  and  t53 
medium 
happening 
before  high 
priority  tests. 

2.5.3 

Medium  to  Low 
Priority 
Relationship 
(T=0) 

The  TE  RSCSP 
model  medium 
priority  test  events 
shall  be  started 
before  low  priority 
test  events. 

The  requirement  shall  be 
verified  by  confirming  that 
schedule  starts  medium 
priority  test  events  before 
low  priority  test  events. 

NOT  VERIFIED 
Model  run  did  not 
include  any  low 
priority  test 
events. 

2.5.4 

High  to  Low 
Priority 
Relationships 
(T=0) 

The  TE  RSCSP 
model  high  priority 
test  events  shall 
be  started  before 
low  priority  test 
events. 

The  requirement  shall  be 
verified  by  confirming  that 
schedule  starts  high  priority 
test  events  before  low 
priority  test  events. 

NOT  VERIFIED 
Model  run  did  not 
include  any  low 
priority  test 
events. 

2.6 

Test  Venue 
Movement  Test 
Periods  (T=0) 

The  TE  RSCSP 
model  shall  be 
constrained  by  the 
test  venue 
movement  test 
periods. 

The  requirement  shall  be 
verified  by  confirming  that 
schedule  uses  the  test 
venue  movement  test 
periods  based  on  the  input 
file. 

MET 

Data  shows  that 
when  moving 
venues,  the  input 
data  matches  the 
input  file. 
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Para  # 

Para  Title 

Requirement 

Verification  Criteria 

Verification  & 
Rationale 

2.6.1 

Add  Time 

Period  from 

Test  Venue 

Movement 

(T=0) 

The  TE  RSCSP 
model  shall  add 
time  periods  to  the 
schedule  based 
on  the  distance 
between  test 

venues. 

The  requirement  shall  be 
verified  by  confirming  that 
schedule  results  show  that 
the  time  periods  added  to 
the  schedule  when  the  test 
asset  moves  to  a  new  test 
venue  are  based  on  the 
input  file. 

MET 

Data  shows  that 
when  moving 
venues,  the  input 
file  test  periods 
are  added  to  the 
test  schedule. 

2.6.2 

Schedule  Test 
Event  on 
Available  Test 
Venues  (T=0) 

The  TE  RSCSP 
model  shall  not 
schedule  a  test 
event  at  a  test 
venue  that  does 
not  perform  that 
test  event. 

The  requirement  shall  be 
verified  by  confirming  that 
schedule  results  show  that 
the  test  events  are 
scheduled  only  on  available 
test  venues  based  on  the 
input  file. 

MET 

Test  venues 
used  track  to  test 
venues  allowed 
based  on  the 
input  file. 

2.7 

Test  Venue 

Test  Asset 
Capacity  (T=0) 

The  TE  RSCSP 
model  shall  be 
constrained  by 
test  venue  test 
asset  capacity 
input. 

The  requirement  shall  be 
verified  by  confirming  that 
the  number  of  test  assets 
located  at  each  test  venue 
on  the  schedule  does  not 
exceed  the  capacity 
identified  in  the  input  file. 

NOT  VERIFIED 
The  number  of 
test  assets 
assigned  to  a 
test  venue  by  the 
model  did  not 
exceed  the 
amount  in  the 

file,  but  the 
numbers  in  the 
file  (10)  did  not 
really  check  the 
model. 


Test  Asset  Test 
Venue  Starting 
Location  (T=0) 

The  TE  RSCSP 
model  shall  be 
constrained  by  the 
test  asset  test 
venue  starting 
location  input. 

The  requirement  shall  be 
verified  by  confirming  that 
schedule  test  assets  start  at 
the  venues  identified  in  the 
test  asset  test  venue  input 
file. 

MET 

The  starting 
locations  and 
quantities  (4  at 
ATC  and  3  at 
AVTB)  track  to 
the  input  file. 

Test  Event  Test 

The  TE  RSCSP 

The  requirement  shall  be 

MET 

Venue 

model  shall  be 

verified  by  confirming  that 

Test  events  are 

Relationships 

constrained  by  the 

schedule  test  events  are 

assigned  to 

(T=0) 

test  event  test 

performed  at  the  test 

allowed  test 

venue 

venues  identified  in  the  test 

venues  based  on 

relationships 

input. 

event  test  venue 
relationships  input  file. 

the  input  file. 

Test  Event 

The  TE  RSCSP 

The  requirement  shall  be 

MET 

Precedence 

model  shall  be 

verified  by  confirming  that 

Data  shows  that 

Relationships 

constrained  by 

schedule  results  are 

predecessor  test 

(T=0) 

test  event 

constrained  by  the  test 

events  in  the 

precedence 

event  precedence 

schedule  occur 

relationships 

input. 

relationships  input  file. 

before  successor 
test  events. 
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Para  #  Para  Title 


Requirement 


Verification  Criteria 


Verification  & 
Rationale 


2.10.1 

Test  Event 
Predecessor 

First  (T=0) 

The  TE  RSCSP 
model  shall 
perform 

predecessor  test 
events  prior  to  test 
event  successor 
test  events. 

The  requirement  shall  be 
verified  by  confirming  that 
predecessor  test  events  are 
on  the  schedule  before 
successor  test  events 
regardless  of  test  event 
priority. 

MET 

Data  shows  that 
predecessor  test 
events  in  the 
schedule  occur 
before  successor 
test  events. 

2.10.2 

Test  Event 
Successor 
Second  (T=0) 

The  TE  RSCSP 
model  shall  not 
perform  successor 
test  events  prior  to 
test  event 
predecessor  test 
events. 

The  requirement  shall  be 
verified  by  confirming  that 
successor  test  events  are 
on  the  schedule  before 
predecessor  test  events 
regardless  of  test  event 
priority. 

MET 

Data  shows  that 
successor  test 
events  in  the 
schedule  occur 
after 

predecessor  test 
events. 

2.11 

Number  of  Test 
Assets  Needed 
for  Test  Events 
(T=0) 

The  TE  RSCSP 
model  shall  use 
the  number  of  test 
assets  needed  for 
test  events  input. 

The  requirement  shall  be 
verified  by  confirming  that 
the  schedule  uses  the 
number  of  test  assets 
needed  for  test  events  input 
file. 

MET 

Data  shows  that 
the  number  of 
test  assets 
needed  for  a  test 
event  are  used. 

For  multi-vehicle 
operations,  tests 
are  performed 
simultaneously 
on  the  test 
assets. 


2.12 

Low  Priority 
Schedule 
Relationship 
(0) 

The  TE  RSCSP 
model  low  priority 
test  events  shall 
be  allowed  to  go 
beyond  the  test 
period. 

The  requirement  shall  be 
verified  by  confirming  that 
schedule  results  match  the 
input  file. 

NOT  VERIFIED 
Low  priority  test 
events  are  not 
used  in  this 
model  run. 

2.13 

Test  Asset 
Unavailability 
(0) 

The  TE  RSCSP 
model  shall  be 
constrained  by  the 
test  asset 
unavailability. 

The  requirement  shall  be 
verified  by  confirming  that 
schedule  places  test  assets 
only  on  available  test  assets 
based  on  the  input  file. 

NOT  VERIFIED 
Data  for  Test 
Asset 

Unavailability  is 
not  included  in 
the  model  run. 

2.14 

Test  Venue 
Unavailability 
(0) 

The  TE  RSCSP 
model  shall  be 
constrained  by 
test  venue 
unavailability. 

The  requirement  shall  be 
verified  by  confirming  that 
schedule  results  are 
constrained  by  the  input  file. 

NOT  VERIFIED 
Data  for  Test 
Venue 

Unavailability  is 
not  included  in 
the  model  run. 

2.15 

Test  Event 
Priority 
Deadlines 
(T=0) 

The  TE  RSCSP 
model  shall  be 
constrained  by  the 
high  and  medium 
test  event  priority 
deadlines. 

The  requirement  shall  be 
verified  by  confirming  that 
schedule  results  are 
constrained  by  the  input  file. 

PARTIALLY 

MET 

This  model  run 
used  190  for  the 
total  test  periods. 
High  is  170,  and 
medium  is  190. 
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Para  # 

Para  Title 

Requirement 

Verification  Criteria 

Verification  & 
Rationale 

2.15.1 

High  Priority  to 

Deadline 

Relationship 

(T=0) 

The  TE  RSCSP 
model  shall 
complete  high 
priority  test  events 
before  high 
priority  deadlines. 

The  requirement  shall  be 
verified  by  confirming  that 
schedule  results  start  high 
priority  test  events  before 
the  high  priority  deadline. 

MET 

2.15.2 

Medium  Priority 
to  Deadline 
Relationship 
(T-O) 

The  TE  RSCSP 
model  shall 
complete  medium 
priority  test  events 
before  medium 
priority  deadlines. 

The  requirement  shall  be 
verified  by  confirming  that 
schedule  results  start 
medium  priority  test  events 
before  the  medium  priority 
deadline. 

NOT  VERIFIED 
This  model  run 
used  190  for  the 
total  test  periods 
and  the  medium 
test  periods, 
which  means 
that  it  is  not 
verified. 

2.15.3 

High  Priority  to 
Test  Period 
Relationship 
(T-O) 

The  TE  RSCSP 
model  shall 
complete  high 
priority  test  events 
before  the  test 
period  completes. 

The  requirement  shall  be 
verified  by  confirming  that 
schedule  results  complete 
high  priority  test  events 
before  the  test  period 
completes. 

MET 

High  priority  test 
events  end  at 

109.  Schedule 
ends  at  test 
period  1 1 1 .  Test 
period  ends  at 

190.  Test  event 
t19,  which 
requires  two 
vehicles,  is  put  at 
the  end. 

2.15.4 

Medium  Priority 
to  Test  Period 
Relationship 
(T=0) 

The  TE  RSCSP 
model  shall 
complete  medium 
priority  test  events 
before  the  test 
period  completes. 

The  requirement  shall  be 
verified  by  confirming  that 
schedule  results  complete 
medium  priority  test  events 
before  the  test  period 
completes. 

MET 

Medium  priority 
test  events  end 
at  test  period 

111.  Schedule 
ends  at  test 
period  111. 

Test  period  ends 
at  190. 

The  results  of  the  model  controls  and  constraints  requirements  schedule 
verification  assessment  indicate  that  the  majority  of  the  threshold  model  controls 
and  constraints  requirements  in  Table  53  are  MET,  except  for  some  that  are 
PARTIALLY  MET  and  some  that  are  NOT  VERIFIED.  There  are  no  controls  and 
constraints  requirements  that  are  verified  to  be  NOT  MET. 

Partially  Met  requirements.  Data  used  for  verification  for  time  period, 
priority  deadlines  did  not  fully  test  the  time  period  due  to  the  values  used.  The 
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following  are  considered  to  be  PARTIALLY  MET,  and  require  an  additional 
verification  event  in  order  to  be  fully  assessed:  One  Test  Event  on  Test  Asset 
during  Time  Period,  Test  Event  Priority  Relationships,  High  Priority  to  Test  Period 
Relationship,  High  to  Medium  Priority  Relationship,  and  Test  Event  Priority 
Deadlines. 

Not  Verified  requirements.  Data  did  not  include  low  priority  test  events, 
test  asset  type,  test  venue  capacity,  test  asset  unavailability,  test  venue 
unavailability,  and  medium  deadline.  The  following  are  considered  to  be  NOT 
VERIFIED,  and  require  an  additional  verification  event  in  order  to  be  assessed: 
Medium  to  Low  Priority  Relationship,  High  to  Low  Priority  Relationships,  Test 
Venue  Test  Asset  Capacity,  Test  Asset  Type  Test  Event  Relationships,  Test 
Asset  Unavailability,  Test  Venue  Unavailability,  Medium  Priority  to  Deadline 
Relationship,  and  Low  Priority  Schedule  Relationship. 
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